feat(kitaru): optional trace source and replay verification

CONTEXT.md

@@ -0,0 +1,143 @@
+# CONTEXT
+
+Shared vocabulary for the Kitaru integration. Product boundary and roadmap
+consequences live in GitHub issue #7; Phase 1 is #8; Phase 2 is #9. Decisions
+are recorded in [`docs/adr/`](docs/adr/).
+
+## Product
+
+**tracegrad** is the evidence-gated optimization engine. It owns the prompt/text
+artifact, artifact lineage and hash, deterministic distillation, attribution,
+failure-theme aggregation, evidence verification, edit synthesis, token-budget
+discipline, rejection memory, human approval, standalone JSONL source,
+post-deployment trends, and the verification summary / decision UX.
+
+**Kitaru** owns provider-specific trace ingestion, trace normalization, the
+session graph, execution storage, evaluators and judge execution, cohorts,
+replay, the worker runtime, tool-history replay, and execution / replay
+visualization.
+
+tracegrad is not an observability or replay platform. Kitaru is not a mandatory
+core dependency. tracegrad does not duplicate Kitaru's execution UI.
+
+```text
+without Kitaru:  proposal → deploy → next batch → trends
+with Kitaru:     proposal → replay verify → deploy → trends
+```
+
+`trends` stays in core. For core-only users it is verification after
+deployment, via the next evaluated batch. For Kitaru users it is post-deployment
+confirmation that replay-verified improvements persist in real traffic.
+
+## Install modes
+
+**Core-only.** `uv tool install tracegrad`. No Kitaru package, server, worker,
+or observability integration. `tracegrad run --traces …`, `apply`, and `trends`
+are unchanged. `import tracegrad` never requires Kitaru.
+
+**tracegrad + Kitaru.** `uv tool install "tracegrad[kitaru]"`, extra pinned
+`kitaru>=0.22,<0.23`. The extra installs a **client**. It does not install
+verification (ADR 0001). Credentials and server URL come from Kitaru's own
+config (`kitaru login`). tracegrad stores no Kitaru secrets. `.tracegradrc`
+holds only non-secret selection (cohort name, evaluation name); CLI flags
+override.
+
+## Source vs batch (Phase 1)
+
+`--source kitaru` is a **fetch-and-map** that writes the JSONL the existing
+pipeline already reads (ADR 0004). `--traces` and `--source kitaru` are
+mutually exclusive. The deterministic core does not learn Kitaru exists. There
+is no `TraceSource` protocol, no `sources/` package, and no `ingest.py` rewrite
+(ADR 0010). All Kitaru SDK code lives under `src/tracegrad/integrations/kitaru/`.
+
+**Session** — a Kitaru recording (`SessionResponse`). Durable id is the UUID;
+`number` is display-only (`#4811`).
+
+**Trace** — a tracegrad `Trace`. One session maps to at most one trace.
+
+**Root LLM node** — an `llm_call` with no `subagent_call` anywhere in its
+ancestry, following `parent_index` **and** `secondary_parent_indexes`. The
+graph is a DAG. A node reachable from a subagent is not root even when one of
+its parents is (ADR 0005). Subagent prompts and tool outputs cannot become the
+artifact or `Trace.output`. Never guess a missing system prompt.
+
+**Source drop** — a Session could not become a Trace. Named kebab-case reasons
+(`system-prompt-unavailable`, `multiple-system-prompts`,
+`judge-rationale-missing`, `judge-score-out-of-range`,
+`judge-score-unsupported`, `judge-score-unavailable`, `output-unavailable`,
+`input-unavailable`, `ambiguous-evaluation`, …).
+
+**Batch drop** — a Trace is not part of this Batch (the four reasons
+`ingest.py` already uses, including `prompt-hash-partition` and
+`rationale-below-quality-floor`).
+
+Source drops and batch drops are two tables, never merged. Merging them lets a
+mapping bug hide behind a legitimate partition.
+
+**Snapshot** — mapped JSONL plus source fingerprint under `.tracegrad/`, written
+before ingest. Re-runs read it; `--refresh` refetches. Cohort version is
+resolved **once** per run (`cohort_version_id` in the fingerprint). `engine=format`
+manifests are refused with a named error (ADR 0002).
+
+**Judge mapping** — ADR 0003. `Score` stays `[0, 1]`; out-of-range and
+non-numeric evaluations drop by name rather than being rescaled.
+`judge_fingerprint` is derived from `evaluator_name` + `evaluator_version`. A
+conflicting manifest value is an error.
+
+## Verification (Phase 2)
+
+`tracegrad verify --backend kitaru --run <id>` reuses the cohort, evaluator,
+and agent metadata Phase 1 persisted. `VerificationBackend` lives in
+`ports.py`; the implementation lives under `integrations/kitaru/`.
+
+**No backend.** `tracegrad verify` prints an actionable message and exits
+non-zero. That does not block `run` / `apply` / `trends`. A verify that exits 0
+having done nothing would read as *verified* in CI.
+
+**Tool policy (hard invariant).** Every tracegrad-created replay sets
+`HistoryConfig(scope=COHORT_VERSION, on_miss=FAIL)`. No passthrough through the
+tracegrad path. A novel call becomes `TOOL_HISTORY_MISS`, not a live production
+side effect.
+
+**Override scope (hard invariant).** Only the root LLM system prompt is
+overridden (`ReplayOverride.system_prompt`). After replay, assert root LLM
+nodes carry the candidate and non-root LLM nodes carry their baseline
+counterpart. Violations are `OVERRIDE_SCOPE_DIVERGENCE`. A failed `select_evaluation`
+is `SELECT_EVALUATION_FAILED` (drop reason in detail). A mismatched
+`evaluator_version` is `EVALUATOR_VERSION_MISMATCH`. Scores that cannot
+be classified are `SCORE_UNCLASSIFIED`. All of these are **incomparable**
+— not improved, not regressed — and stay in the per-session buckets.
+Headline aggregates still come from Kitaru (ADR 0006).
+
+**Cohort constraint.** Mixed-agent-version cohorts are refused with a
+per-version breakdown (ADR 0007). Baseline and candidate use the same evaluator
+version.
+
+**Apply gate.** With a backend configured (the originating run persisted Kitaru
+source metadata), `apply` refuses unless a persisted verification exists whose
+`candidate_prompt_hash` equals the hash of what is about to be written.
+`--force` overrides. Core-only users are unaffected (ADR 0009).
+
+**Persistence / resume.** `.tracegrad/verification/<verification-id>.json`.
+Persist the Kitaru run id immediately after creation. An interrupted
+verification resumes and watches the existing experiment run rather than
+creating a duplicate.
+
+**Inspection.** Hand off the dashboard base plus real identifiers. Do not build
+`--open` or `tracegrad inspect` (ADR 0008).
+
+**Verdict.** The verification report never prints `SHIP`, and never applies or
+reverts on its own. The human decides.
+
+**Aggregates.** Headline numbers come from
+`/api/v1/ui/experiment-runs/{id}/evaluation-aggregates` so they match the
+Kitaru UI. That `/api/v1/ui/` namespace is a UI-support contract, contained by
+the `<0.23` pin.
+
+## Do not build
+
+Langfuse / LangSmith / Braintrust importers. A tracegrad-native replay engine.
+Worker infrastructure. Tool mocking or a history engine. Cohort storage. Replay
+experiment orchestration. A full eval runner. A competing trace viewer.
+`--open` / `inspect`. An `approve`/`apply` rename (deferred, ADR 0009).
+Multi-artifact editing (README "Beyond system prompts" is unaffected).

README.md

@@ -78,9 +78,16 @@ tracegrad init
 tracegrad run --traces batch.jsonl --manifest manifest.json --estimate  # cost preview
 tracegrad run --traces batch.jsonl --manifest manifest.json             # analyze, propose
 tracegrad apply                                                         # review, accept/reject
+tracegrad trends                                                        # last two runs
 tracegrad status                                                        # budget, trends, ledgers
 ```
 
+Core-only install has no Kitaru package, server, worker, or observability
+integration. `run --traces`, `apply`, and `trends` work without it.
+`tracegrad verify` without a backend prints an actionable message and exits
+non-zero — a verify that exits 0 having done nothing would read as verified
+in CI.
+
 `run` prints one review card per proposed edit — the diff, the verbatim quotes
 behind it, and any flags — and writes the proposal to `.tracegrad/`. Nothing
 touches your prompt until `tracegrad apply`. `apply --revert` restores the
@@ -94,6 +101,68 @@ the last two runs.
 Attribution is one model call per trace, so `run --jobs 8` is worth setting on
 any batch above a handful of traces.
 
+### Optional: Kitaru as a trace source and replay backend
+
+Kitaru is an **optional extra**, pinned `kitaru>=0.22,<0.23`. The base package
+has no Kitaru dependency; `import tracegrad` never requires it. The extra
+installs a **client**. It does not install verification — replays run on a
+worker in *your* agent virtualenv. tracegrad stores no Kitaru secrets; use
+`kitaru login`.
+
+```sh
+uv tool install "tracegrad[kitaru]"
+# from this repo until PyPI:
+#   uv tool install "git+https://github.com/dnth/tracegrad[kitaru]"
+kitaru login
+```
+
+`.tracegradrc` may hold only non-secret selection; flags override:
+
+```toml
+[kitaru]
+cohort = "support-production"
+evaluation = "quality"
+```
+
+`--source kitaru` is a fetch-and-map: it writes JSONL the existing pipeline
+already reads, then ingest runs unchanged. `--traces` and `--source kitaru`
+are mutually exclusive. `engine = "format"` manifests are refused with a
+named error. Mapped traces and a source fingerprint are snapshotted under
+`.tracegrad/sources/kitaru/` before ingest; re-runs read the snapshot;
+`--refresh` refetches. The cohort version is resolved once per run.
+
+```sh
+tracegrad run \
+  --source kitaru \
+  --kitaru-cohort support-production \
+  --kitaru-evaluation quality \
+  --manifest manifest.json
+```
+
+`judge_fingerprint` is derived from the evaluator (`quality@3`). A conflicting
+manifest value is an error. Set the manifest fingerprint to that derived
+identity.
+
+After a proposal, verify the candidate against the same frozen cohort. This
+needs a running Kitaru server, a worker in the agent's virtualenv, and a
+registered agent version:
+
+```sh
+tracegrad verify --backend kitaru --run run-0001
+tracegrad apply --all
+```
+
+`apply` then refuses unless a persisted verification exists whose
+`candidate_prompt_hash` matches what is about to be written. `--force`
+overrides. Core-only JSONL users are unaffected. The verification report
+never prints `SHIP` and never applies or reverts on its own.
+
+Every tracegrad-created replay sets recorded tool history with `on_miss=fail`.
+Passthrough is not reachable through the tracegrad path.
+
+`trends` stays in core either way: without Kitaru it is the next-batch check
+after deploy; with Kitaru it confirms a replay-verified change in real traffic.
+
 ### Try it on the bundled example
 
 `example/` holds a synthetic 13-trace batch for a support agent, its manifest,
@@ -163,8 +232,10 @@ tracegrad reads an optional TOML file named `.tracegradrc` from the project root
 If it is absent, `neverDelete = []`, `minEffect = 0.05`, `minCoverage = 0.8`,
 and `convergenceRuns = 2` apply. The default attribution and synthesis harness
 providers are `openai` and `claude`. The supported top-level keys are
-`neverDelete`, `minEffect`, `minCoverage`, `convergenceRuns`, and
-`harness_presets`; see the package configuration model for the preset fields.
+`neverDelete`, `minEffect`, `minCoverage`, `convergenceRuns`,
+`harness_presets`, and `kitaru`; see the package configuration model for the
+preset fields. The `[kitaru]` table holds only non-secret selection (`cohort`,
+`evaluation`); credentials stay in `kitaru login`.
 
 ```toml
 neverDelete = ["prompt/identity"]
@@ -200,11 +271,13 @@ measured it.
 
 ```
 .tracegrad/
-  distilled/    content-addressed distilled traces — the only text a quote may cite
-  ledgers/      append-only JSONL: runs, gaps, rejections, applied edits
-  reports/      per-run theme counts, the input to trend comparison
-  runs/         per-run proposal, resume checkpoint, and autopsy of dropped proposals
-  snapshots/    the prompt as it was before each apply
+  distilled/      content-addressed distilled traces — the only text a quote may cite
+  ledgers/        append-only JSONL: runs, gaps, rejections, applied edits
+  reports/        per-run theme counts, the input to trend comparison
+  runs/           per-run proposal, resume checkpoint, and autopsy of dropped proposals
+  snapshots/      the prompt as it was before each apply
+  sources/        optional mapped JSONL from `--source kitaru`, plus the source fingerprint
+  verification/   persisted replay-verification state (resume-safe)
 ```
 
 Everything except `apply` only reads and appends. A killed run resumes from its

docs/adr/0001-tracegrad-submits-verification-it-does-not-host-it.md

@@ -0,0 +1,49 @@
+# 0001. tracegrad submits verification; it does not host it
+
+## Status
+
+Accepted
+
+## Context
+
+Kitaru replay re-executes the user's real agent code. `ExperimentRunCreateRequest`
+requires an `agent_version_id`, and `AgentVersionResponse` carries a `RunSpec`
+with a shell command. Kitaru's README is explicit that replays run "on workers
+in your environment: your virtualenv, your credentials, your network."
+
+tracegrad is installed via `uv tool install`, into an isolated virtualenv, which
+by construction is not the virtualenv the agent runs in. The optional extra
+`tracegrad[kitaru]` installs a client library. It does not install a worker, a
+server, Docker, or the user's agent.
+
+Four capabilities an earlier draft assumed do not exist in the Kitaru 0.22 API,
+and treating the extra as "gaining replay verification" would hide the real
+prerequisites.
+
+## Decision
+
+tracegrad submits verification; it does not host it.
+
+Phase 2 requires the user to already have:
+
+1. a running Kitaru server (FastAPI + Postgres, via Docker),
+2. a worker process running in the virtualenv where their agent code lives,
+3. their agent instrumented with a Kitaru adapter and registered as an agent
+   version.
+
+`tracegrad verify` preflights those before spending anything: probe the server,
+confirm a live worker claims this agent version (workers report `last_seen_at`),
+and confirm the agent version and cohort version resolve. A replay experiment
+is paid and slow; "no worker is polling" should surface in milliseconds.
+
+tracegrad never hosts a worker and does not try to.
+
+## Consequences
+
+- The extra is a client pin (`kitaru>=0.22,<0.23`), not a verification runtime.
+- Core-only users keep a complete JSONL workflow with no Kitaru package, server,
+  or worker.
+- Documentation must not claim that `tracegrad[kitaru]` "gains replay
+  verification" by itself.
+- Preflight cannot check whether an adapter honours a system-prompt override;
+  that gap is closed by ADR 0006's post-replay assertion, not by prediction.

docs/adr/0002-prompt-lineage-from-recorded-prompts.md

@@ -0,0 +1,44 @@
+# 0002. Prompt lineage comes from recorded system prompts
+
+## Status
+
+Accepted
+
+## Context
+
+tracegrad's ingest partitions a batch on `prompt_hash`. Rates are only
+meaningful within one prompt version, so only the dominant partition survives.
+The hash has to name the artifact under study.
+
+Kitaru sessions do not carry a tracegrad prompt hash. They do carry per-node
+`system_prompt_selector` pointers into node inputs. Those recorded prompts are
+the only honest lineage: guessing a missing system prompt would attribute
+failures to an artifact the session never ran.
+
+`engine = "format"` manifests render differently per request. Hashing the
+recorded (already rendered) prompts of a format template would give N partitions
+of size one, and ingest would keep a batch of one. Reverse-templating those
+recordings back onto a format template is a separate problem and is not solved
+here.
+
+## Decision
+
+Hash the recorded system prompt extracted from root LLM nodes. The mapped batch
+must be single-valued on that hash, which is what ingest already enforces.
+
+This supports `engine = "none"` only. When the manifest declares
+`engine = "format"`, refuse the Kitaru source with a named error rather than
+proceeding into a batch that collapses.
+
+Never infer or guess a missing system prompt. Zero unique recorded prompts is
+`system-prompt-unavailable`. More than one unique recorded prompt on the root
+LLM nodes of one session is `multiple-system-prompts`.
+
+Reverse-templating for `format` prompts is deferred, not rejected.
+
+## Consequences
+
+- Kitaru-sourced runs with a `format` manifest fail closed.
+- `prompt_hash` on a mapped `Trace` is `text_hash` of the extracted system
+  prompt, so the rest of the pipeline does not learn Kitaru exists.
+- Multi-prompt sessions drop at the source rather than poisoning the partition.