|
1 | 1 | # ICP face recognition |
2 | 2 |
|
3 | | -This is an ICP smart contract runs face detection and face recognition of user's photo that can be uploaded either from a camera or a local file. |
| 3 | +This is an ICP smart contract that runs face detection and face recognition on user photos that can be uploaded either from a camera or a local file. |
4 | 4 |
|
5 | 5 | The smart contract consists of two canisters: |
6 | 6 |
|
7 | | -- The backend canister embeds the [the Tract ONNX inference engine](https://github.com/sonos/tract) with two ONNX models. One model is used to detect a face in the photo and return its bounding box. Another model is used for computing face embeddings. |
8 | | -- The frontend canister contains the Web assets such as HTML, JS, CSS that are served to the browser. |
| 7 | +- The **backend canister** embeds the [Tract ONNX inference engine](https://github.com/sonos/tract) with two ONNX models. One model is used to detect a face in the photo and return its bounding box. Another model is used for computing face embeddings. |
| 8 | +- The **frontend canister** contains the Web assets such as HTML, JS, and CSS that are served to the browser. |
9 | 9 |
|
10 | | -# Models |
| 10 | +## Models |
11 | 11 |
|
12 | 12 | The smart contract uses two models: one for detecting the face and another for recognizing the face. |
13 | 13 |
|
14 | | -## Face detection |
| 14 | +Since the models are large, they cannot be embedded into the Wasm binary of the smart contract. Instead they must be uploaded separately after deployment. |
15 | 15 |
|
16 | | -A face detection model finds the bounding box of a face in the image. |
17 | | -You can download [Ultraface](https://github.com/onnx/models/tree/main/validated/vision/body_analysis/ultraface) - ultra-lightweight face detection model - [[here](https://github.com/onnx/models/blob/bec48b6a70e5e9042c0badbaafefe4454e072d08/validated/vision/body_analysis/ultraface/models/version-RFB-320.onnx)]. |
| 16 | +### Face detection |
18 | 17 |
|
19 | | -Alternatively, you can run |
| 18 | +A face detection model finds the bounding box of a face in the image. |
| 19 | +You can download [Ultraface](https://github.com/onnx/models/tree/main/validated/vision/body_analysis/ultraface) — ultra-lightweight face detection model — by running: |
20 | 20 |
|
21 | | -``` |
| 21 | +```bash |
22 | 22 | ./download-face-detection-model.sh |
23 | 23 | ``` |
24 | 24 |
|
25 | | -## Face recognition |
| 25 | +### Face recognition |
26 | 26 |
|
27 | 27 | A face recognition model computes a vector embedding of an image with a face. |
28 | 28 | You can obtain a pretrained model from [facenet-pytorch](https://github.com/timesler/facenet-pytorch) as follows. |
29 | 29 |
|
30 | | -- #### Step 1: Install `python` and `pip`: https://packaging.python.org/en/latest/tutorials/installing-packages/. |
| 30 | +1. Install `python` and `pip`: https://packaging.python.org/en/latest/tutorials/installing-packages/ |
31 | 31 |
|
32 | | -- #### Step 2: Install `facenet-pytorch` and `torch`: |
| 32 | +2. Install `facenet-pytorch`, `torch`, and `onnx`: |
33 | 33 |
|
34 | | -``` |
35 | | -pip install facenet-pytorch |
36 | | -pip install torch |
37 | | -pip install onnx |
38 | | -``` |
| 34 | + ```bash |
| 35 | + pip install facenet-pytorch |
| 36 | + pip install torch |
| 37 | + pip install onnx |
| 38 | + ``` |
39 | 39 |
|
40 | | -- #### Step 3: Export ONNX model. Start a python shell and run the following commands or create a python file and run it: |
| 40 | +3. Export the ONNX model. Start a Python shell and run: |
41 | 41 |
|
42 | | -``` |
43 | | -import torch |
44 | | -import facenet_pytorch |
45 | | -resnet = facenet_pytorch.InceptionResnetV1(pretrained='vggface2').eval() |
46 | | -input = torch.randn(1, 3, 160, 160) |
47 | | -torch.onnx.export(resnet, input, "face-recognition.onnx", verbose=False, opset_version=11) |
48 | | -``` |
| 42 | + ```python |
| 43 | + import torch |
| 44 | + import facenet_pytorch |
| 45 | + resnet = facenet_pytorch.InceptionResnetV1(pretrained='vggface2').eval() |
| 46 | + input = torch.randn(1, 3, 160, 160) |
| 47 | + torch.onnx.export(resnet, input, "face-recognition.onnx", verbose=False, opset_version=11) |
| 48 | + ``` |
49 | 49 |
|
50 | | -- #### Step 4: This should produce `face-recognition.onnx`. Copy the file to the root of this repository. |
| 50 | + This produces `face-recognition.onnx`. Copy the file to the root of this repository. |
51 | 51 |
|
52 | | -## Prerequisites |
| 52 | +## Build and deploy from the command line |
53 | 53 |
|
54 | | -- [x] Install the [IC |
55 | | - SDK](https://internetcomputer.org/docs/current/developer-docs/getting-started/install). For local testing, `dfx >= 0.22.0` is required. |
56 | | -- [x] Clone the example dapp project: `git clone https://github.com/dfinity/examples` |
57 | | -- [x] Install `wasi2ic`: Follow the steps in https://github.com/wasm-forge/wasi2ic and make sure that `wasi2ic` binary is in your `$PATH`. |
58 | | -- [x] Install `wasm-opt`: `cargo install wasm-opt` |
| 54 | +### Prerequisites |
59 | 55 |
|
60 | | -## Build the application |
| 56 | +- Node.js |
| 57 | +- icp-cli: `npm install -g @icp-sdk/icp-cli @icp-sdk/ic-wasm` |
61 | 58 |
|
62 | | -``` |
63 | | -dfx start --background |
64 | | -dfx deploy |
| 59 | +### Install |
| 60 | + |
| 61 | +```bash |
| 62 | +git clone https://github.com/dfinity/examples |
| 63 | +cd examples/rust/face-recognition |
65 | 64 | ``` |
66 | 65 |
|
67 | | -If the deployment is successful, the it will show the `frontend` URL. |
68 | | -Open that URL in browser to interact with the smart contract. |
| 66 | +### Deploy and test |
69 | 67 |
|
70 | | -## Chunk uploading of models |
| 68 | +```bash |
| 69 | +icp network start -d |
| 70 | +icp deploy |
| 71 | +make test |
| 72 | +icp network stop |
| 73 | +``` |
71 | 74 |
|
72 | | -Since the models are large, they cannot be embedded into the Wasm binary of the smart contract. |
73 | | -Instead they should be uploaded separately. |
| 75 | +Run `npm run dev` from the `frontend/` directory for hot reload during frontend development. |
74 | 76 |
|
75 | | -[DecideAI](https://decideai.xyz/) implemented a tool for incremental uploading of models: https://github.com/modclub-app/ic-file-uploader/tree/main. |
| 77 | +### Upload the ONNX models |
76 | 78 |
|
77 | | -You can install the tool with |
| 79 | +After deploying, upload the ONNX models to the canister using [ic-file-uploader](https://github.com/modclub-app/ic-file-uploader/tree/main): |
78 | 80 |
|
79 | | -``` |
| 81 | +```bash |
80 | 82 | cargo install ic-file-uploader |
| 83 | +make upload-models |
81 | 84 | ``` |
82 | 85 |
|
83 | | -Afterwards, execute the `upload-models-to-canister.sh` script, which runs the following commands: |
84 | | - |
85 | | -``` |
86 | | -dfx canister call backend clear_face_detection_model_bytes |
87 | | -dfx canister call backend clear_face_recognition_model_bytes |
88 | | -ic-file-uploader backend append_face_detection_model_bytes version-RFB-320.onnx |
89 | | -ic-file-uploader backend append_face_recognition_model_bytes face-recognition.onnx |
90 | | -dfx canister call backend setup_models |
91 | | -``` |
| 86 | +Once the models are uploaded, open the frontend URL in your browser to interact with the smart contract. |
92 | 87 |
|
93 | 88 | ## Credits |
94 | 89 |
|
95 | 90 | Thanks to [DecideAI](https://decideai.xyz/) for discussions and providing [ic-file-uploader](https://github.com/modclub-app/ic-file-uploader/tree/main). |
96 | 91 |
|
| 92 | +## Security considerations and best practices |
| 93 | + |
| 94 | +See the [ICP security best practices](https://docs.internetcomputer.org/guides/security/overview). |
0 commit comments