feat: add opt-in Jev audits and citation repair

.env.example

@@ -13,6 +13,18 @@ RAG_EVAL_MODEL=qwen3:8b
 RAG_OPENAI_BASE_URL=http://localhost:1234/v1
 RAG_OPENAI_API_KEY=
 
+# Optional hosted citation audit. Off keeps all inference local.
+# Observer mode sends generated claims and their cited excerpts to TypeSafe.
+# Repair mode also checks other retrieved excerpts and may replace weak citations.
+# Modes: off, observe, repair. Claim text is preserved in repair mode.
+RAG_JEV_MODE=off
+TYPESAFE_API_KEY=
+RAG_JEV_MODEL=jev-1.13.0
+# Total budget per question part, not per individual claim.
+RAG_JEV_TIMEOUT=5
+# Experimental policy threshold; calibrate against reviewed claims.
+RAG_JEV_THRESHOLD=0.90
+
 # Retrieval and storage.
 RAG_DATA_DIR=data/qdrant
 RAG_COLLECTION=documents

README.md

@@ -87,6 +87,39 @@ The browser interface is organized as an evidence desk: manage and select source
 
 The application stores vectors beneath `data/qdrant`. Both `.env` and `data/` are ignored by Git.
 
+## Optional Jev experiment
+
+Jev can check whether each generated claim follows from its actual cited excerpts. It runs after
+structured generation, independently of the chat and embedding providers. Enable the observer in
+your local `.env`:
+
+```dotenv
+RAG_JEV_MODE=observe
+TYPESAFE_API_KEY=your-local-key
+```
+
+Restart the app with `uv run --env-file .env local-rag`. The answer now includes a **Claim checks**
+panel with support, contradiction, and insufficient-evidence probabilities, a suggested action,
+and expandable audit details. At the default experimental threshold of 0.90, Python suggests
+`keep` for high support, `withhold` for high contradiction or insufficient evidence, and `review`
+otherwise. These suggestions leave the answer and citation ledger unchanged. Provider failures
+show `unavailable` and still return the original answer.
+
+For opt-in citation repair, set `RAG_JEV_MODE=repair` and restart. When a claim's original check
+favors insufficient evidence, the app checks alternative passages already retrieved for that
+question part, one at a time in retrieval order. It replaces the citation with the first passage
+whose fresh support check meets the threshold. Claim text and the retrieved ledger stay intact.
+Existing support, contradiction, and unavailable results are retained. If no replacement passes
+or the shared audit budget expires, the original citation remains. Details retain the original
+assessment and every attempted replacement.
+
+Observation sends generated claims and their cited excerpt text to TypeSafe's hosted API.
+Repair also sends alternative retrieved excerpts when testing replacements. Each candidate is
+checked in isolation; passages from other question parts and filenames are excluded.
+The default remains `off`, with no Jev network calls. Review your documents before enabling it.
+See the [design study](docs/jev-integration-study.md), [API notes](docs/jev-api-research.md), and
+[measured tests and replay instructions](docs/jev-testing.md).
+
 ## API
 
 Upload a document:
@@ -122,7 +155,9 @@ The response keeps generation and retrieval separately inspectable:
       "score": 0.82,
       "source_sha256": "0123456789abcdef..."
     }
-  ]
+  ],
+  "audit_mode": "off",
+  "audits": []
 }
 ```
 
@@ -132,6 +167,14 @@ output and retrieved passages is disabled before rendering. Fenced flow diagrams
 PDFs are presented to both the reader and grounded generator as readable steps while their
 original citation `text` remains unchanged.
 
+With Jev enabled, `audit_mode` is `observe` or `repair` and `audits` contains one record per claim.
+Its citation IDs refer to the numbered response ledger. Records include the validated Choice
+distribution, suggested action, model, threshold, elapsed time, excerpt hashes, and request usage.
+Claims sharing the same cited set share a request ID and its token counts; deduplicate request IDs
+when summing usage. Repair records add `original`, `repair_attempts`, and `repair_policy_version`;
+deduplicate request IDs across those nested checks and the final record too. No-claim abstentions
+produce an empty audit list. `/health` also exposes `jev_mode`.
+
 Use `POST /api/retrieve` with the same request body to inspect retrieval without generation. Delete one document with `DELETE /api/documents/{document_id}`, or clear the collection with `DELETE /api/documents`. A full clear and re-index is required after changing embedding models because vectors from different embedding spaces cannot be mixed.
 
 ## Swap providers independently
@@ -173,6 +216,11 @@ To swap embeddings instead, change `RAG_EMBEDDING_PROVIDER` and `RAG_EMBEDDING_M
 | `RAG_SCORE_THRESHOLD` | `0.15` | minimum score for the dense retrieval leg; BM25 ranks independently |
 | `RAG_MAX_UPLOAD_MB` | `10` | upload boundary |
 | `RAG_REQUEST_TIMEOUT` | `120` | model request timeout in seconds |
+| `RAG_JEV_MODE` | `off` | `observe` for hosted checks; `repair` also verifies replacement citations |
+| `TYPESAFE_API_KEY` | empty | secret used only for Jev requests |
+| `RAG_JEV_MODEL` | `jev-1.13.0` | pinned Jev model |
+| `RAG_JEV_TIMEOUT` | `5` | total audit budget per question part in seconds; at most three parts |
+| `RAG_JEV_THRESHOLD` | `0.90` | experimental action threshold; greater than 0.5 and at most 1 |
 
 ## Project layout
 
@@ -214,6 +262,7 @@ Development follows `feature/* → dev → main`. `dev` is the default integrati
 - Answers must cover each requested part from cited context, explicitly identify unsupported parts, and avoid filling gaps with model background knowledge.
 - Upload extension and size are validated, and uploaded filenames are never used as filesystem destinations.
 - Choosing a remote compatible endpoint changes the privacy boundary. Review that provider before sending documents.
+- Enabling Jev sends claims and excerpts to TypeSafe; repair mode may also send alternative retrieved passages. Its judgments do not establish factual correctness, answer completeness, or injection safety.
 - This demo has no authentication and binds to `127.0.0.1`. Do not expose it directly to a network.
 
 See [SECURITY.md](SECURITY.md) for reporting and deployment guidance.
@@ -228,7 +277,10 @@ See [SECURITY.md](SECURITY.md) for reporting and deployment guidance.
 | GitHub Actions and Pages | $0 within the public-repository allowances |
 | Cloud infrastructure | not used |
 
-Pulumi, AWS, hosted model APIs, authentication, background workers, neural reranking, and multi-user tenancy are deferred. They add cost or operational weight without improving this local portfolio baseline. The [architecture](https://sanjaysrinivas.github.io/context-foundry/) lists the concrete triggers for each upgrade.
+The optional Jev observer uses a paid hosted API; it is outside the zero-cost local default.
+Pulumi, AWS, authentication, background workers, neural reranking, and multi-user tenancy are
+deferred. The [architecture](https://sanjaysrinivas.github.io/context-foundry/) lists the concrete
+triggers for each upgrade.
 
 ## Evaluation and limitations
 

SECURITY.md

@@ -28,5 +28,17 @@ Before any shared deployment, add and verify:
 
 The default Ollama configuration keeps inference local. Selecting an OpenAI-compatible remote URL sends prompts or document chunks to that endpoint; its privacy and retention policy then applies.
 
+`RAG_JEV_MODE=observe` independently enables TypeSafe's hosted Jev API. It sends generated claims
+and the text of their actual cited excerpts, including sensitive content if it occurs in those
+excerpts. Observation excludes uncited passages from the request state. `RAG_JEV_MODE=repair`
+also submits alternative passages already retrieved for the same question part, individually,
+when looking for replacement citations. Claim text and the retrieved ledger are preserved;
+citations change only after a fresh support check passes the threshold. Filenames, credentials,
+and passages retrieved only for other question parts remain excluded from each check.
+Keep `TYPESAFE_API_KEY` in the Git-ignored `.env` or environment. Audit records expose sanitized
+status errors rather than remote error bodies. They are rendered as plain text in the browser.
+Observation does not filter answers. Citation repair does not rewrite claims or establish that
+the answer is true, complete, or relevant; Jev judgments are not a security boundary.
+
 Retrieved documents are untrusted input. The application includes prompt-injection guidance, but model instructions are not a security boundary. Never grant model output authority to execute commands or perform external actions without separate validation and authorization.
 

docs/evaluation.md

@@ -177,6 +177,28 @@ regressions reproducible but means acceptable paraphrases must be represented by
 phrases or reviewed manually. The gates are initial portfolio targets, not universal production
 guarantees.
 
+## Optional Jev claim/citation replay
+
+The Jev observer has a separate replay runner and eight public synthetic development fixtures in
+`evaluation/jev-smoke.jsonl`. It checks a claim against its cited passages directly, without
+Ollama, ingestion, or retrieval. Uncited decoys test evidence isolation. This diagnostic does not
+alter the approved-gold release metrics or promote synthetic labels to gold.
+
+~~~powershell
+uv run --frozen --env-file .env python -m local_rag.jev_evaluation --output evaluation/results/jev-smoke.json
+~~~
+
+The command sends the fixture claims and cited text to TypeSafe and requires `TYPESAFE_API_KEY`.
+It retains distributions and provenance, and exits with status 1 if any expected label disagrees
+or an audit is unavailable. A disagreement is evidence to inspect, not a reason to rewrite the
+expected label. The default 0.90 action threshold is experimental; reported confidence is not an
+independent probability of correctness. See [live results and limitations](jev-testing.md).
+
+For an enforcement experiment, first add source-reviewed claim/citation labels to a frozen,
+representative holdout and compare unsupported acceptance, supported withholding, review and
+outage rates, latency, and cost. The existing ledger-based citation metric alone does not measure
+whether each displayed claim follows from its particular citations.
+
 ## Change protocol
 
 1. Freeze the corpus, source hashes, evaluation cases, model tags, and retrieval configuration.