Skip to content

Ditto MCP server — knowledge RAG + credential-passthrough action tools - #2522

Open
kalinkostashki wants to merge 11 commits into
eclipse-ditto:masterfrom
boschglobal:ditto-mcp-server
Open

Ditto MCP server — knowledge RAG + credential-passthrough action tools#2522
kalinkostashki wants to merge 11 commits into
eclipse-ditto:masterfrom
boschglobal:ditto-mcp-server

Conversation

@kalinkostashki

@kalinkostashki kalinkostashki commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds an extensible Ditto MCP server (TypeScript, @modelcontextprotocol/sdk) under mcp/. It gives AI agents (e.g. Claude Code) two capabilities over one server:

  1. Knowledge / RAG — searchable Ditto documentation (public llms.txt + optional local corpus) via keyword, semantic, or hybrid retrieval.
  2. Ditto action tools — auto-generated from the OpenAPI spec, with credential passthrough (the MCP forwards the caller's credentials; Ditto authorizes) and a ToolPolicy (read-only by default; writes and sudo/devops are opt-in).

All new code lives in mcp/; no existing files are modified. 31 test files; default suite is hermetic (Postgres tests are opt-in behind test:pg + Docker).

What's included (by commit / phase)

  • Foundation — Zod-validated config + loader, core ToolDef/ToolRegistry, ping, server factory, stdio and streamable-HTTP transports (loopback bind, session store, DNS-rebinding protection), request-context threading.
  • Knowledge core (FTS) — chunker, SQLite FTS5 retriever, PublicSource (llms.txt), KnowledgeService + search/get_chunk tools, one shared service across sessions.
  • Semantic + hybrid retrieval — local ONNX bge-small embeddings, sqlite-vec store, HybridRetriever (reciprocal-rank fusion), local-dir corpus, config-selectable fts/vector/hybrid, lazy ONNX load, markdown-only ingest, bounded-concurrency fetch, batched embeddings (bounds memory), retrieval provenance.
  • Persistent knowledge index — file-backed KnowledgeStore (chunks + FTS + vectors), ingest CLI, atomic ingest (temp + rename), index-metadata validation, async store lifecycle + openStore factory, PgKnowledgeStore (pgvector + Postgres tsvector) with testcontainers tests.
  • Ditto action tools + auth — OpenAPI parser + operation-to-tool generation with a pinned bundled spec fallback, HttpDittoClient, credential passthrough (basic / devops / OIDC client-credentials; config-level and per-session Authorization header), ToolPolicy (allow-methods + write allowlist + spec-security-aware sudo gating for /devops, /sudo, /connections), shallow typed request-body schemas.
  • Documentation + OSS packaging — consolidated mcp/README.md, five example configs, hardened .gitignore.

Configuration scenarios (see mcp/examples/)

  • public-fts.json — OSS default: public docs, keyword search, no Ditto.
  • hybrid-local.json — hybrid RAG over public docs + a local corpus.
  • pgvector.json — shared Postgres/pgvector index.
  • ditto-readonly.json — docs + read-only (GET) Ditto tools.
  • ditto-oidc-write.json — OIDC creds + selected write tools.

Testing

Verified end-to-end against a live k3d Ditto (basic ditto:ditto, devops devops:devops):

  • Ingested the full public corpus → 1033 chunks (hybrid, bge, metadata complete).
  • tools/list under a GET-only policy exposed zero write/sudo/connection tools (policy filtering works).
  • Semantic search returned relevant MQTT/reconnect docs (FTS + vector both firing).
  • Live Ditto GET via an MCP tool → HTTP 200.
  • Allow-listed PUT (create thing) → HTTP 201, confirmed independently.
  • Sudo /connections with a non-devops credential → refused; with a devops credential → gate passes, Ditto returns 200.
  • Passthrough over HTTP transport, driven by Claude Code as the MCP client: server configured with no stored credentials, Authorization header supplied per request → thing list/search reached Ditto using the forwarded credentials only.

Security notes

  • The MCP performs no authorization itself — it forwards credentials and lets Ditto decide.
  • Read-only is the default; writes and sudo/devops operations are opt-in via explicit allowlists.
  • Credentials are never logged. Internal corpora are protected by deployment isolation, not by MCP-side auth.
  • HTTP transport binds loopback by default with DNS-rebinding protection on.

How to run

cd mcp && npm install && npm run build
DITTO_MCP_CONFIG=examples/public-fts.json node dist/bin/ingest.js # build the index
DITTO_MCP_CONFIG=examples/public-fts.json node dist/bin/stdio.js # or dist/bin/http.js

kalinkostashki and others added 10 commits August 10, 2026 14:32
@kalinkostashki @claude
feat(mcp): foundation — config schema, core registry, ping, stdio + H…
431b707 
…TTP transports

Zod-validated AppConfig + loader, ToolDef/ToolRegistry core, ping tool, server factory, stdio and streamable-HTTP entrypoints (loopback bind, session store, DNS-rebinding protection), request-context threading.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kalinkostashki @claude
feat(mcp): knowledge core — FTS5 keyword retrieval over Ditto docs
e2ebb39 
Chunk/KnowledgeSource/Retriever types, markdown chunker, SQLite FTS5 retriever, PublicSource llms.txt loader, KnowledgeService + search/get_chunk tools wired via config, shared async service across sessions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kalinkostashki @claude
feat(mcp): semantic + hybrid retrieval — bge embeddings, sqlite-vec, …
9dab958 
…RRF fusion

Local ONNX bge-small embeddings, sqlite-vec store + VectorRetriever, HybridRetriever (reciprocal rank fusion), LocalDirSource corpus, config-selectable fts/vector/hybrid, lazy ONNX load, markdown-only ingest, bounded-concurrency fetch, batched embeddings (OOM fix), limit param + provenance.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kalinkostashki @claude
feat(mcp): persistent knowledge index — file-backed store, ingest CLI…
469d2e3 
…, pgvector

File-backed KnowledgeStore (chunks+FTS5+vectors), read-only retrievers over the store, ingest CLI, atomic ingest (temp+rename), index metadata validation, async store lifecycle + openStore factory, PgKnowledgeStore (pgvector + tsvector) with testcontainers pg tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kalinkostashki @claude
feat(mcp): Ditto action tools — swagger-gen, credential passthrough, …
f6653df 
…ToolPolicy

OpenAPI operation parser + operation-to-tool generation with pinned bundled spec, HttpDittoClient, credential passthrough (basic/devops/OIDC client-credentials, config + per-session), ToolPolicy (read-only default, write allowlist, spec-security-aware sudo gating), typed request-body schemas.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kalinkostashki @claude
docs(mcp): documentation + OSS packaging
d4c8d4b 
Consolidated README, example configs, hardened .gitignore; docs for HTTP/config, knowledge/persistence/pgvector, action tools/passthrough/policy, OIDC + typed bodies.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kalinkostashki @claude
feat(mcp): devops credential + dynamic openapi spec source
44c85c0 
Squashes the devops-credential and openapi-spec-source work.

Devops credential:
- Add optional `ditto.devopsCredential` used exclusively for
  sudo-classified operations (/devops/*, sudo*, and the secret-bearing
  /api/2/connections* API); all other operations use `ditto.credential`.
- Sudo ops are refused at the MCP layer when no devopsCredential is set.
  Connectivity is classified sudo via a spec-independent path rule so it
  is always devops-gated.
- Remove the `devops` credential kind and the `isDevops` flag: devops
  capability is now positional (which credential slot). A per-session
  Authorization header overrides the selected credential without
  inheriting devops status. Credential kind is now `basic | oidc` only.

OpenAPI spec source:
- Resolve the spec by precedence path > url > version > in-repo canonical;
  add `openApi.version` (git tag/ref) and `openApi.versionUrlTemplate`.
- Default fallback reads the canonical in-repo spec
  (documentation/src/main/resources/openapi/ditto-api-2.yml) instead of a
  committed duplicate; drop mcp/assets/ditto-openapi.yml.

BREAKING CHANGE: kind:"devops" and the devops:true flag are removed; move a
devops credential into ditto.devopsCredential. Sudo operations now require
ditto.devopsCredential to be set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kalinkostashki @claude
feat(mcp): Docker deployment + configurable chunking
be04178 
- Dockerfile (multistage node:22-slim, glibc), .dockerignore, and
  docker/config.docker.json: HTTP transport, non-root, native deps baked in.
  Secure-by-default DNS rebinding protection + loopback allowedHosts.
- bin shebangs so stdio/http/ingest run as installed CLIs.
- Lazy-load sqlite/pg store backends and local embeddings; friendly errors
  naming the missing native dep instead of raw MODULE_NOT_FOUND.
- knowledge.chunk.{maxChars,overlap} config wired through makeSources to
  PublicSource + LocalDirSource (was hardcoded to chunker defaults).
- README: Docker Deployment and Chunking sections (incl. bge-small
  512-token ceiling for vector/hybrid).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kalinkostashki @claude
feat(mcp): neighbor expansion for knowledge search
a4a2361 
Retriever-agnostic context expansion in KnowledgeService.search: around each
anchor, pull ±context same-document chunks (same cite, adjacent ordinal) by id
— never re-scored or re-embedded. Results are deduped and emitted as contiguous
spans ordered by best anchor rank; neighbors are tagged role="context" so their
relevance isn't over-weighted vs. matches.

- Stops at document boundaries (different cite) even when ordinals are globally
  contiguous, so expansion never leaks across docs.
- knowledge.search.{limit,context} config (defaults 5 / 1); search tool gains a
  per-call `context` arg that overrides. `limit` counts anchors; neighbors extra.
- Works for fts/vector/hybrid: an anchor's neighbors are defined by document
  layout, not by how it was matched — so expansion is embedding-neutral and
  sidesteps the 512-token ceiling.
- README: Search (query-time) section incl. overlap-vs-expansion interaction.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kalinkostashki
add eclipse license where appropriate
82b6715
@kalinkostashki kalinkostashki changed the title Draft: Ditto MCP server — knowledge RAG + credential-passthrough action tools Ditto MCP server — knowledge RAG + credential-passthrough action tools Aug 18, 2026
@kalinkostashki
fix: create friendlier configs for writeAllowlist
b4d6bca 
- removed writeAllowlist underscores from configs
- stripped unnecessary underscores from tools generation
- updated documentation to reflect the changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kalinkostashki