diff --git a/README.md b/README.md index b37f6c6..3d93abe 100644 --- a/README.md +++ b/README.md @@ -159,10 +159,13 @@ To swap embeddings instead, change `RAG_EMBEDDING_PROVIDER` and `RAG_EMBEDDING_M | `RAG_CHAT_PROVIDER` | `ollama` | chat adapter | | `RAG_EMBEDDING_PROVIDER` | `ollama` | embedding adapter | | `RAG_OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama API root | +| `RAG_OPENAI_BASE_URL` | `http://localhost:1234/v1` | OpenAI-compatible API root | +| `RAG_OPENAI_API_KEY` | empty | optional compatible-endpoint API key | | `RAG_CHAT_MODEL` | `llama3.2:3b` | generation model | | `RAG_EMBEDDING_MODEL` | `embeddinggemma` | embedding model | | `RAG_EVAL_MODEL` | `qwen3:8b` | optional local Ragas judge; should reliably follow tool schemas | | `RAG_DATA_DIR` | `data/qdrant` | persistent vector-store path | +| `RAG_COLLECTION` | `documents` | Qdrant collection name | | `RAG_CHUNK_SIZE` | `900` | characters per chunk | | `RAG_CHUNK_OVERLAP` | `150` | repeated characters between chunks | | `RAG_EMBEDDING_BATCH_SIZE` | `32` | chunks embedded per provider request | diff --git a/docs/index.html b/docs/index.html index 9663492..1e22257 100644 --- a/docs/index.html +++ b/docs/index.html @@ -133,7 +133,7 @@

System context

-

Two explicit pipelines

+

Three explicit pipelines

Ingestion

@@ -158,6 +158,16 @@

Question answering

  • Render Markdown and citation markers deterministically, then return the retrieved proof.
    • +
    +

    Evaluation data

    +
      +
    1. Generate optional direct-fact synthetic silver candidates from stable source evidence.
      2. +
    2. Validate source support, answer support, schema integrity, and duplicate questions.
      3. +
    3. Review the source first, then approve, edit, or reject each candidate.
      4. +
    4. Freeze only approved_gold cases into a versioned release dataset.
      5. +
    5. Run deterministic retrieval, grounding, citation, and abstention gates; use Ragas only as a secondary diagnostic.
      6. +
    +
    @@ -207,7 +217,7 @@

    Provider seams

    Data and retrieval model

    Vector

    The embedding is stored with cosine distance. Collection dimensionality is discovered from the first embedding and checked thereafter.

    -

    Payload

    document_id, source, page, chunk_index, and text support citations, document filtering, deletion, and replacement on re-ingestion.

    +

    Payload

    document_id, source_sha256, source, page, chunk_index, and the actual chunk text support stable evidence anchors, citations, hybrid retrieval, document filtering, deletion, and replacement on re-ingestion.

    Hybrid rank

    BM25Plus ranks stored chunk text while Qdrant ranks embeddings. Reciprocal-rank fusion combines positions, not incompatible raw score magnitudes.

    Chunking

    PyMuPDF4LLM preserves PDF layout as Markdown; RapidOCR handles scanned pages locally; LangChain's standalone splitter preserves headings and natural text boundaries while enforcing configurable character limits.

    Consistency rule

    Changing the embedding model or provider requires rebuilding the collection. Mixing vector spaces produces invalid similarity results.

    diff --git a/evaluation/cases.example.jsonl b/evaluation/cases.example.jsonl index e5f2ea2..f058aa9 100644 --- a/evaluation/cases.example.jsonl +++ b/evaluation/cases.example.jsonl @@ -1,2 +1,2 @@ -{"case_id":"notes-storage-001","dataset_version":"1.0.0","split":"smoke","corpus_id":"notes-demo","corpus_version":"1","question":"Where is the evidence stored?","question_type":"direct_fact","answerable":true,"expected_evidence":[{"source":"notes.txt","page":1,"contains":["stored locally"],"evidence_group":"storage"}],"required_facts":["stored locally"],"reference_answer":"The evidence is stored locally.","document_ids":[],"generation_method":"human","review_status":"approved_gold","tags":["direct_fact"]} +{"case_id":"notes-storage-001","dataset_version":"1.0.0","split":"smoke","corpus_id":"notes-demo","corpus_version":"1","question":"Where is the evidence stored?","question_type":"direct_fact","answerable":true,"expected_evidence":[{"source":"notes.txt","source_sha256":"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef","page":1,"contains":["stored locally"],"evidence_group":"storage"}],"required_facts":["stored locally"],"reference_answer":"The evidence is stored locally.","document_ids":[],"generation_method":"human","review_status":"approved_gold","tags":["direct_fact"]} {"case_id":"notes-absent-001","dataset_version":"1.0.0","split":"smoke","corpus_id":"notes-demo","corpus_version":"1","question":"Who approved the evidence?","question_type":"unanswerable","answerable":false,"expected_evidence":[],"required_facts":[],"reference_answer":null,"document_ids":[],"generation_method":"human","review_status":"approved_gold","tags":["unanswerable"]}