Skip to content

Commit 45e6c56

Browse files
authored
chore: migrate motoko/llm_chatbot to icp-cli (#1441)
1 parent 3dff4f3 commit 45e6c56

14 files changed

Lines changed: 333 additions & 244 deletions

File tree

‎.github/workflows/llm_chatbot.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ on:
55
branches: [master]
66
pull_request:
77
paths:
8+
- motoko/llm_chatbot/**
89
- rust/llm_chatbot/**
910
- .github/workflows/llm_chatbot.yml
1011

@@ -13,6 +14,19 @@ concurrency:
1314
cancel-in-progress: true
1415

1516
jobs:
17+
motoko-llm_chatbot:
18+
runs-on: ubuntu-24.04
19+
container: ghcr.io/dfinity/icp-dev-env-motoko:1.0.1
20+
env:
21+
ICP_CLI_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
22+
steps:
23+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
24+
- name: Deploy
25+
working-directory: motoko/llm_chatbot
26+
run: |
27+
icp network start -d
28+
icp deploy
29+
1630
rust-llm_chatbot:
1731
runs-on: ubuntu-24.04
1832
container: ghcr.io/dfinity/icp-dev-env-rust:1.0.1

‎motoko/llm_chatbot/.devcontainer/devcontainer.json‎

Lines changed: 0 additions & 20 deletions
This file was deleted.

‎motoko/llm_chatbot/BUILD.md‎

Lines changed: 0 additions & 113 deletions
This file was deleted.

‎motoko/llm_chatbot/README.md‎

Lines changed: 53 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,46 +1,79 @@
11
# LLM Chatbot
22

3-
The LLM Chatbot example demonstrates how an ICP smart contract can be used to interact with a large language model (LLM) to generate text. The user can input a prompt, and the smart contract will use the LLM to generate a response.
4-
The response is then returned to the user, and the user can submit some follow-up prompts to continue the conversation.
3+
This example demonstrates how an ICP canister can interact with a large language model (LLM) to generate text. The user can input a prompt and the canister will use the LLM to generate a response. Follow-up prompts continue the conversation with the full message history.
54

6-
## Deploying from ICP Ninja
5+
## How it works
76

8-
When viewing this project in ICP Ninja, you can deploy it directly to the mainnet for free by clicking "Run" in the upper right corner. Open this project in ICP Ninja:
7+
The backend canister calls the [LLM canister](https://forum.dfinity.org/t/introducing-the-llm-canister-deploy-ai-agents-with-a-few-lines-of-code/41424)'s `v1_chat` endpoint directly (see `backend/app.mo`), without a helper library. It reads the LLM canister's principal from the `PUBLIC_CANISTER_ID:llm` environment variable. Locally, `icp deploy` deploys a copy of the LLM canister (backed by Ollama) and injects this variable automatically. On mainnet the shared LLM canister already exists, so `icp.yaml` sets the variable to its principal (`w36hm-eqaaa-aaaal-qr76a-cai`) for the `ic` environment.
98

10-
[![](https://icp.ninja/assets/open.svg)](https://icp.ninja/i?g=https://github.com/dfinity/examples/motoko/llm_chatbot)
9+
## Build and deploy from the command line
1110

12-
## Build and deploy from the command-line
11+
### Prerequisites
1312

14-
### 1. [Download and install the IC SDK.](https://internetcomputer.org/docs/building-apps/getting-started/install)
13+
- Node.js
14+
- icp-cli: `npm install -g @icp-sdk/icp-cli @icp-sdk/ic-wasm`
15+
- ic-mops: `npm install -g ic-mops`
1516

16-
### 2. Setting up Ollama
17+
### Set up Ollama (local deployment only)
1718

18-
To be able to test the agent locally, you'll need a server for processing the agent's prompts. For that, we'll use `ollama`, which is a tool that can download and serve LLMs.
19-
See the documentation on the [Ollama website](https://ollama.com/) to install it. Once it's installed, run:
19+
The LLM canister delegates inference to [Ollama](https://ollama.com/). Install it and then run:
2020

21-
```
21+
```bash
2222
ollama serve
23-
# Expected to start listening on port 11434
2423
```
2524

26-
The above command will start the Ollama server, so that it can process requests by the agent. Additionally, and in a separate window, run the following command to download the LLM that will be used by the agent:
25+
In a separate terminal, download the model (about 4 GiB, one-time) and load it
26+
into memory:
27+
28+
```bash
29+
ollama run llama3.1:8b "hi"
30+
```
31+
32+
`ollama run` pulls the model if needed and warms it in memory. This matters: the
33+
LLM canister's HTTP outcall to Ollama has a ~30 s deadline, and a cold model
34+
load alone can take longer than that — so the *first* call after `ollama serve`
35+
starts may time out (`SysFatal: Timeout expired`) if the model isn't warm yet.
36+
Warming it first avoids this; `ollama serve` then keeps it loaded. If you do hit
37+
a timeout on the first call, simply retry — the model stays resident afterwards.
38+
39+
### Install
2740

41+
```bash
42+
git clone https://github.com/dfinity/examples
43+
cd examples/motoko/llm_chatbot
2844
```
29-
ollama run llama3.1:8b
45+
46+
### Deploy
47+
48+
```bash
49+
icp network start -d
50+
icp deploy
3051
```
3152

32-
The above command will download an 8B parameter model, which is around 4GiB. Once the command executes and the model is loaded, you can terminate it. You won't need to do this step again.
53+
Open the frontend URL printed by `icp deploy` to use the chatbot in the browser. Make sure Ollama is running with the model warmed (see above) so the first message does not time out.
3354

34-
### 3. Download your project from ICP Ninja using the 'Download files' button on the upper left corner, or [clone the GitHub examples repository.](https://github.com/dfinity/examples/)
55+
For hot-reload frontend development:
3556

36-
### 4. Navigate into the project's directory.
57+
```bash
58+
npm run dev --prefix frontend
59+
```
3760

38-
### 5. Deploy the project to your local environment:
61+
## Deploying to mainnet
3962

63+
```bash
64+
icp deploy -e ic
4065
```
41-
dfx start --background --clean && dfx deploy
66+
67+
No Ollama setup is needed — mainnet calls go directly to the LLM canister at `w36hm-eqaaa-aaaal-qr76a-cai`.
68+
69+
## Updating the Candid interface
70+
71+
The `backend/backend.did` file defines the backend canister's public interface. The frontend TypeScript bindings are auto-generated from this file during the frontend build. If you modify the backend's public API, regenerate the `.did` file:
72+
73+
```bash
74+
mops generate candid backend
4275
```
4376

4477
## Security considerations and best practices
4578

46-
If you base your application on this example, it is recommended that you familiarize yourself with and adhere to the [security best practices](https://internetcomputer.org/docs/building-apps/security/overview) for developing on ICP. This example may not implement all the best practices.
79+
If you base your application on this example, familiarize yourself with the [security best practices](https://docs.internetcomputer.org/guides/security/overview) for developing on ICP. This example may not implement all best practices.

‎motoko/llm_chatbot/backend/app.mo‎

Lines changed: 78 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,85 @@
1-
import LLM "mo:llm";
1+
import Runtime "mo:core/Runtime";
22

3-
persistent actor {
4-
public func prompt(prompt : Text) : async Text {
5-
await LLM.prompt(#Llama3_1_8B, prompt);
3+
// The backend calls the LLM canister's `v1_chat` endpoint directly. The request
4+
// and response types below mirror that canister's Candid interface:
5+
// https://dashboard.internetcomputer.org/canister/w36hm-eqaaa-aaaal-qr76a-cai#interface
6+
//
7+
// LLM canister interface:
8+
// v1_chat : (chat_request_v1) -> (chat_response_v1)
9+
//
10+
// This example does not use tools, so the optional `tools` field is omitted from
11+
// the request — Candid decodes an absent optional record field as `null` on the
12+
// canister side, keeping the call wire-compatible.
13+
actor {
14+
15+
// A message in a chat. Mirrors `chat_message_v1` in the LLM canister interface.
16+
public type ChatMessage = {
17+
#user : { content : Text };
18+
#system_ : { content : Text };
19+
#assistant : AssistantMessage;
20+
#tool : { content : Text; tool_call_id : Text };
21+
};
22+
23+
// The assistant's reply. Mirrors `assistant_message`.
24+
public type AssistantMessage = {
25+
content : ?Text;
26+
tool_calls : [ToolCall];
27+
};
28+
29+
public type ToolCall = {
30+
id : Text;
31+
function : FunctionCall;
32+
};
33+
34+
public type FunctionCall = {
35+
name : Text;
36+
arguments : [ToolCallArgument];
37+
};
38+
39+
public type ToolCallArgument = {
40+
name : Text;
41+
value : Text;
42+
};
43+
44+
// Request/response for `v1_chat`. The request omits the optional `tools` field
45+
// (see the note above).
46+
type Request = {
47+
model : Text;
48+
messages : [ChatMessage];
49+
};
50+
type Response = { message : AssistantMessage };
51+
type LlmActor = actor { v1_chat : (Request) -> async Response };
52+
53+
// The model this example uses. The LLM canister identifies models by string;
54+
// other available models include "qwen3:32b" and "llama4-scout".
55+
let model = "llama3.1:8b";
56+
57+
// The LLM canister ID is injected as PUBLIC_CANISTER_ID:llm at deploy time:
58+
// local: auto-injected by icp-cli after deploying the pre-built llm canister
59+
// ic: set explicitly in icp.yaml to the shared mainnet LLM canister
60+
//
61+
// See icp.yaml for the environment configuration.
62+
func llmCanister<system>() : LlmActor {
63+
let ?id = Runtime.envVar<system>("PUBLIC_CANISTER_ID:llm") else Runtime.trap("PUBLIC_CANISTER_ID:llm not set — run icp deploy");
64+
actor (id) : LlmActor;
65+
};
66+
67+
func sendChat<system>(messages : [ChatMessage]) : async AssistantMessage {
68+
let response = await llmCanister<system>().v1_chat({ model; messages });
69+
response.message;
670
};
771

8-
public func chat(messages : [LLM.ChatMessage]) : async Text {
9-
let response = await LLM.chat(#Llama3_1_8B).withMessages(messages).send();
72+
public func prompt(prompt : Text) : async Text {
73+
let message = await sendChat<system>([#user({ content = prompt })]);
74+
switch (message.content) {
75+
case (?text) text;
76+
case null "";
77+
};
78+
};
1079

11-
switch (response.message.content) {
80+
public func chat(messages : [ChatMessage]) : async Text {
81+
let message = await sendChat<system>(messages);
82+
switch (message.content) {
1283
case (?text) text;
1384
case null "";
1485
};
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
type ToolCallArgument =
2+
record {
3+
name: text;
4+
value: text;
5+
};
6+
type ToolCall =
7+
record {
8+
function: FunctionCall;
9+
id: text;
10+
};
11+
type FunctionCall =
12+
record {
13+
arguments: vec ToolCallArgument;
14+
name: text;
15+
};
16+
type ChatMessage =
17+
variant {
18+
assistant: AssistantMessage;
19+
system: record {content: text;};
20+
tool: record {
21+
content: text;
22+
tool_call_id: text;
23+
};
24+
user: record {content: text;};
25+
};
26+
type AssistantMessage =
27+
record {
28+
content: opt text;
29+
tool_calls: vec ToolCall;
30+
};
31+
service : {
32+
chat: (messages: vec ChatMessage) -> (text);
33+
prompt: (prompt: text) -> (text);
34+
}

0 commit comments

Comments
 (0)