Skip to content

feat: add opt-in Jev audits and citation repair - #31

Merged
Sanjaysrinivas merged 1 commit into
devfrom
feature/jev-observer
Oct 1, 2026
Merged

Sanjaysrinivas merged 1 commit into
devfrom
feature/jev-observer

Conversation

@Sanjaysrinivas

Copy link
Copy Markdown
Owner

Summary

  • A correct claim can carry a citation that does not establish it. Add opt-in Jev checks and citation repair: the live NIST AES example keeps “16 bytes” and replaces citation [1] with [2], which received 0.97 support.
  • Keep the default local workflow with RAG_JEV_MODE=off. observe reports judgments without changing the answer; repair changes citations only after a fresh check passes the experimental 0.90 support threshold.

Changes

  • Call the TypeSafe API directly with pinned jev-1.13.0, strict response validation, isolated cited-excerpt sets, sanitized errors, and an optional API key kept out of settings representations.
  • Try single replacement passages from the same question part's retrieval results only for claims judged insufficient. Initial checks and repair attempts share a five-second budget; claim text stays unchanged. Preserve original checks, replacement attempts, evidence hashes, request IDs, model versions, and reported usage.
  • Expose audits in /api/query and the UI, including privacy disclosures and repair provenance. Constrain mobile grid tracks so wide PDF tables scroll within passages.
  • Add 41 mocked Jev tests, eight public synthetic replay fixtures, API/document research, a fixture-only HTML experiment, and measured testing notes. No new dependencies.

Verification

  • uv run --frozen nox — lint, strict type checking, and 93 tests passed; coverage 85.59%. Provider tests use HTTP mocks and require no API keys or downloaded models.
  • Relevant browser/API flow verified — real Ollama/Qdrant/Jev AES repair, unchanged supported MIXCOLUMNS citation, live Jev rejection of a seeded false throughput claim, and Firefox desktop/mobile checks. The 390-pixel page and expanded trace have no horizontal overflow; off, empty, unavailable, retained-citation, and literal-HTML rendering paths passed.
  • Retrieval/parser/model changes include before/after golden-dataset metrics, or not applicable — existing parsing, retrieval, answer-model configuration, and answer-generation prompt are unchanged. The new hosted judge has separate development diagnostics; these are not approved-gold release metrics.

The live public Jev replay agreed with 7/8 expected labels, with zero unsupported acceptances and zero supported withholds. The disagreement remains documented rather than relabeled. The public-document and seeded-negative checks are small development experiments, not threshold calibration or evidence of general correctness. Jev can support individual claims while an answer remains incomplete; this change does not rewrite or remove unsupported claims.

Details: implementation and measured tests.

Screenshots / UX

Hand-authored synthetic UI previews, not live model output. Full live desktop/mobile captures remain local with the ignored test outputs.

Before citation repair (observer):

Synthetic observer preview

After a verified replacement:

Synthetic citation repair preview

Checklist

  • Documentation updated when behavior or architecture changed.
  • No private documents, evaluation cases, model output, secrets, or local vector data committed. Public synthetic fixtures and summarized public-document experiments are documented; raw live responses, downloaded PDFs, credentials, and vector data remain ignored.
  • Breaking changes and migration steps documented, or not applicable — optional modes default to off; API fields are additive. Enabling hosted checks requires TYPESAFE_API_KEY and an app restart.

@Sanjaysrinivas
Sanjaysrinivas merged commit e5fbac6 into dev Oct 1, 2026
8 checks passed
@Sanjaysrinivas
Sanjaysrinivas deleted the feature/jev-observer branch October 1, 2026 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant