|
1 | 1 | # LLM Chatbot |
2 | 2 |
|
3 | | -The LLM Chatbot example demonstrates how an ICP smart contract can be used to interact with a large language model (LLM) to generate text. The user can input a prompt, and the smart contract will use the LLM to generate a response. |
4 | | -The response is then returned to the user, and the user can submit some follow-up prompts to continue the conversation. |
| 3 | +This example demonstrates how an ICP canister can interact with a large language model (LLM) to generate text. The user can input a prompt and the canister will use the LLM to generate a response. Follow-up prompts continue the conversation with the full message history. |
5 | 4 |
|
6 | | -## Deploying from ICP Ninja |
| 5 | +## How it works |
7 | 6 |
|
8 | | -When viewing this project in ICP Ninja, you can deploy it directly to the mainnet for free by clicking "Run" in the upper right corner. Open this project in ICP Ninja: |
| 7 | +The backend canister calls the [LLM canister](https://forum.dfinity.org/t/introducing-the-llm-canister-deploy-ai-agents-with-a-few-lines-of-code/41424)'s `v1_chat` endpoint directly (see `backend/app.mo`), without a helper library. It reads the LLM canister's principal from the `PUBLIC_CANISTER_ID:llm` environment variable. Locally, `icp deploy` deploys a copy of the LLM canister (backed by Ollama) and injects this variable automatically. On mainnet the shared LLM canister already exists, so `icp.yaml` sets the variable to its principal (`w36hm-eqaaa-aaaal-qr76a-cai`) for the `ic` environment. |
9 | 8 |
|
10 | | -[](https://icp.ninja/i?g=https://github.com/dfinity/examples/motoko/llm_chatbot) |
| 9 | +## Build and deploy from the command line |
11 | 10 |
|
12 | | -## Build and deploy from the command-line |
| 11 | +### Prerequisites |
13 | 12 |
|
14 | | -### 1. [Download and install the IC SDK.](https://internetcomputer.org/docs/building-apps/getting-started/install) |
| 13 | +- Node.js |
| 14 | +- icp-cli: `npm install -g @icp-sdk/icp-cli @icp-sdk/ic-wasm` |
| 15 | +- ic-mops: `npm install -g ic-mops` |
15 | 16 |
|
16 | | -### 2. Setting up Ollama |
| 17 | +### Set up Ollama (local deployment only) |
17 | 18 |
|
18 | | -To be able to test the agent locally, you'll need a server for processing the agent's prompts. For that, we'll use `ollama`, which is a tool that can download and serve LLMs. |
19 | | -See the documentation on the [Ollama website](https://ollama.com/) to install it. Once it's installed, run: |
| 19 | +The LLM canister delegates inference to [Ollama](https://ollama.com/). Install it and then run: |
20 | 20 |
|
21 | | -``` |
| 21 | +```bash |
22 | 22 | ollama serve |
23 | | -# Expected to start listening on port 11434 |
24 | 23 | ``` |
25 | 24 |
|
26 | | -The above command will start the Ollama server, so that it can process requests by the agent. Additionally, and in a separate window, run the following command to download the LLM that will be used by the agent: |
| 25 | +In a separate terminal, download the model (about 4 GiB, one-time) and load it |
| 26 | +into memory: |
| 27 | + |
| 28 | +```bash |
| 29 | +ollama run llama3.1:8b "hi" |
| 30 | +``` |
| 31 | + |
| 32 | +`ollama run` pulls the model if needed and warms it in memory. This matters: the |
| 33 | +LLM canister's HTTP outcall to Ollama has a ~30 s deadline, and a cold model |
| 34 | +load alone can take longer than that — so the *first* call after `ollama serve` |
| 35 | +starts may time out (`SysFatal: Timeout expired`) if the model isn't warm yet. |
| 36 | +Warming it first avoids this; `ollama serve` then keeps it loaded. If you do hit |
| 37 | +a timeout on the first call, simply retry — the model stays resident afterwards. |
| 38 | + |
| 39 | +### Install |
27 | 40 |
|
| 41 | +```bash |
| 42 | +git clone https://github.com/dfinity/examples |
| 43 | +cd examples/motoko/llm_chatbot |
28 | 44 | ``` |
29 | | -ollama run llama3.1:8b |
| 45 | + |
| 46 | +### Deploy |
| 47 | + |
| 48 | +```bash |
| 49 | +icp network start -d |
| 50 | +icp deploy |
30 | 51 | ``` |
31 | 52 |
|
32 | | -The above command will download an 8B parameter model, which is around 4GiB. Once the command executes and the model is loaded, you can terminate it. You won't need to do this step again. |
| 53 | +Open the frontend URL printed by `icp deploy` to use the chatbot in the browser. Make sure Ollama is running with the model warmed (see above) so the first message does not time out. |
33 | 54 |
|
34 | | -### 3. Download your project from ICP Ninja using the 'Download files' button on the upper left corner, or [clone the GitHub examples repository.](https://github.com/dfinity/examples/) |
| 55 | +For hot-reload frontend development: |
35 | 56 |
|
36 | | -### 4. Navigate into the project's directory. |
| 57 | +```bash |
| 58 | +npm run dev --prefix frontend |
| 59 | +``` |
37 | 60 |
|
38 | | -### 5. Deploy the project to your local environment: |
| 61 | +## Deploying to mainnet |
39 | 62 |
|
| 63 | +```bash |
| 64 | +icp deploy -e ic |
40 | 65 | ``` |
41 | | -dfx start --background --clean && dfx deploy |
| 66 | + |
| 67 | +No Ollama setup is needed — mainnet calls go directly to the LLM canister at `w36hm-eqaaa-aaaal-qr76a-cai`. |
| 68 | + |
| 69 | +## Updating the Candid interface |
| 70 | + |
| 71 | +The `backend/backend.did` file defines the backend canister's public interface. The frontend TypeScript bindings are auto-generated from this file during the frontend build. If you modify the backend's public API, regenerate the `.did` file: |
| 72 | + |
| 73 | +```bash |
| 74 | +mops generate candid backend |
42 | 75 | ``` |
43 | 76 |
|
44 | 77 | ## Security considerations and best practices |
45 | 78 |
|
46 | | -If you base your application on this example, it is recommended that you familiarize yourself with and adhere to the [security best practices](https://internetcomputer.org/docs/building-apps/security/overview) for developing on ICP. This example may not implement all the best practices. |
| 79 | +If you base your application on this example, familiarize yourself with the [security best practices](https://docs.internetcomputer.org/guides/security/overview) for developing on ICP. This example may not implement all best practices. |
0 commit comments