Skip to content
Open
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,6 @@ dist-ssr
# other secret files
.env.local
.env.langfuse

# local models config
local-models.json
46 changes: 46 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,52 @@ npm run dev
NGROK_URL="<NGROK URL>" # Optional local Supabase Storage tunnel for provider-readable signed URLs
```

### 3. Optional Local LLM Models

To add your own local models (LM Studio, Ollama, LiteLLM, etc.) alongside the cloud models:

1. Start one or more local OpenAI-compatible servers (for example LM Studio on port `1234`).
2. Copy the example catalog and edit entries to match your servers:

```bash
cp local-models.json.example local-models.json
```

Each entry needs an `id`, `name`, `description`, and `baseUrl`. The `id` must match what your server exposes.

Example:

```json
{
"id": "qwen2.5-vl-7b-instruct",
"name": "Qwen2.5 VL 7B",
"description": "Local VLM",
"baseUrl": "http://localhost:1234",
"supportsTools": true,
"supportsVision": true,
"useForAux": true
},
{
"id": "deepseek-v4-flash",
"name": "DeepSeek v4 Flash",
"description": "Third Party LLM",
"baseUrl": "https://api.deepseek.com",
"apiKey": "DEEPSEEK_API_KEY",
"supportsTools": true,
"supportsThinking": true,
"supportsVision": false,
"useForAux": false
}
```

3. Set local model API keys in `.env.local`.
- `apiKey` in `local-models.json` should be the env var name, not the secret value.
- Example: `"apiKey": "DEEPSEEK_API_KEY"` and then set `DEEPSEEK_API_KEY="<your key>"` in `.env.local`.

4. Restart `npm run dev`. Models with a `baseUrl` appear in the parametric model picker.

Cloud models (Gemini, Claude, OpenAI, Z-AI) keep using their existing cloud routes. Only ids listed in `local-models.json` with a `baseUrl` are sent to that model's server.

## 🌐 Setting Up ngrok for Local Development

CADAM uses public URLs for provider callbacks and local signed storage URLs:
Expand Down
23 changes: 23 additions & 0 deletions local-models.json.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[
{
"id": "qwen2.5-vl-7b-instruct",
"name": "Qwen2.5 VL 7B",
"description": "Local VLM",
"baseUrl": "http://localhost:1234",
"supportsTools": true,
"supportsThinking": false,
"supportsVision": true,
"useForAux": true
},
{
"id": "deepseek-v4-flash",
"name": "DeepSeek v4 Flash",
"description": "Third Party LLM",
"baseUrl": "https://api.deepseek.com",
"apiKey": "DEEPSEEK_API_KEY",
"supportsTools": true,
"supportsThinking": true,
"supportsVision": false,
"useForAux": false
},
]
Comment thread
nathan-t4 marked this conversation as resolved.
Loading