diff --git a/CONTEXT.md b/CONTEXT.md index 78858dd..82d1f94 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -297,6 +297,15 @@ must all be present and pass aggregate, query-form, and repository guardrails. F diagnostic and inherit that decision. The last grouped fold is the explicit untouched final test and must pass independently. Artifacts retain exact blocker values, deterministic grouped bootstrap intervals, and fold-level selection/local-perturbation stability diagnostics. +Schema 26 adds binary NDCG@5/10/20/50 without changing the corpus or Production retrieval. Unique chunks +matching at least one resolved exact file-plus-symbol target receive gain one; duplicates and chunks +matching multiple targets cannot add repeated gain, and zero ideal gain returns NDCG zero. The direct +objective selects NDCG@5 first and is compared with a `direct-recall-first` ablation under identical +folds, seeds, and guardrails. Recall@20, Recall@50, and ContextRecall@4096 remain hard direct-retrieval +coverage guardrails; reranker objectives remain recall-first with the full Recall@5/10/20/50 and +ContextRecall@4096 guardrail set. Empty authored ground truth is rejected by the manifest schema. The +Recall-first direct ablation selects from the same archive without changing the three-scenario beam +budget, and both proxy-promotion and successive-halving perform objective-specific archive selection. ### Scorer diff --git a/benchmarks/BASELINE.md b/benchmarks/BASELINE.md index 4eb97f6..666bbd3 100644 --- a/benchmarks/BASELINE.md +++ b/benchmarks/BASELINE.md @@ -1,7 +1,26 @@ # Preliminary Retrieval Baseline The schema-17 entries below are historical artifacts from the benchmark-owned Sparse implementation. -Current schema-24 runs use the production SparseEmbedder and IndexStore without benchmark vector caches. +Current schema-26 runs use the production SparseEmbedder and IndexStore without benchmark vector caches. + +## Schema 26: Binary NDCG Direct-Objective Smoke + +The `fd` MiniLM smoke artifact `retrieval-2026-08-06T18-27-13.604Z.json` compares the new NDCG@5-first +`direct` objective with the historical `direct-recall-first` priority under the same grouped folds, +seeds, DBSF fusion, and hard Recall@20/50 plus Context@4k guardrails. The artifact is local and ignored. + +| Objective | NDCG@5 | NDCG@10 | NDCG@20 | NDCG@50 | R@5 | R@20 | Context@4k | +| -------------- | -----: | ------: | ------: | ------: | ----: | ----: | ---------: | +| NDCG@5-first | 57.0% | 61.5% | 62.5% | 64.2% | 73.3% | 91.3% | 87.3% | +| Recall@5-first | 59.8% | 63.2% | 64.2% | 65.8% | 76.7% | 91.3% | 87.3% | + +This single-repository smoke run validates the complete measurement and ablation path, but does not +support promoting the NDCG-first candidate: Recall-first is stronger on every reported holdout NDCG +cutoff and R@5 while tying the coverage guardrails. A full multi-repository run is still required for +any Production decision; issue #172 itself remains benchmark-only. +The matched Successive-Halving smoke artifact `retrieval-2026-08-06T18-27-50.835Z.json` also completed +with objective-specific archive selection, validating that the ablation is executed rather than emitted +as repeated labels in both supported search strategies. ## Schema 24: Selectable Router Search diff --git a/benchmarks/README.md b/benchmarks/README.md index c863ac9..31b9d22 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -9,7 +9,7 @@ surface. The suite answers five questions independently: 1. Do identity, CamelCase, BM25, dense, and sparse retrieval each contribute useful candidates? -2. Does production-weighted RRF improve `Recall@K` over individual channels and combinations? +2. Does the production DBSF configuration improve `Recall@K` and binary `NDCG@K` over individual channels and combinations? 3. Does full RRF reduce the quality gap between a small general embedder and a code-specific one? 4. How much authored ground truth fits into fixed context budgets? 5. Can one evidence-based router outperform static weights without knowing the authored query form? @@ -53,6 +53,12 @@ representations remain in the same validation fold. Gold targets are exact `file Matching uses extracted identifiers or exact declaration syntax inside the specified file; loose symbol-name segment matching is not accepted. +The existing corpus needs no NDCG-specific labels. Every resolved chunk matching one or more exact +gold targets has binary gain `1`; non-matching chunks have gain `0`. A chunk that resolves multiple +targets and a duplicate occurrence of the same chunk each contribute at most once. NDCG is `0` when +no gold target resolves. Manifest decoding rejects empty ground-truth arrays, and corpus preparation +rejects authored targets that do not resolve to a chunk. + | Corpus | Revision | Language | Size band | Indexed scope | | --------- | ------------------------------------------ | ---------- | --------- | ----------------------------- | | FastAPI | `95f8322ee1dcda7ceace7b1c4f6c9915b36d748f` | Python | medium | `fastapi/**/*.py` | @@ -95,7 +101,7 @@ vp run bench:retrieval:full `bench:retrieval` aliases `bench:retrieval:validate`. Every profile measures the same physical rankings and retrieval variants; profiles only control matrix size, holdout coverage, and expensive -diagnostics. The selected profile is recorded in schema-24 artifacts without changing retrieval +diagnostics. The selected profile is recorded in schema-26 artifacts without changing retrieval semantics. The full profile includes all three fusion methods; short profiles intentionally omit RRF to keep development runs fast. @@ -158,8 +164,8 @@ defaulting to MiniLM. Select another with `PIX_BENCH_MODELS`. Supported values a The router search defaults to `proxy-promotion`. Set `PIX_BENCH_ROUTER_STRATEGY` to `successive-halving` to select the historical Successive-Halving variant. It uses the original lexicographic `R@20`, `R@10`, `Context@4k`, and MRR comparator plus its `halvingKeepFactor`. -Both strategies use the same candidate evaluator and native worker queue, so their artifacts can be -compared directly. +Both strategies use the same candidate evaluator and native worker queue. Their final archive selection +is objective-specific, so Direct and Reranker rows are real comparisons rather than repeated labels. The Jina code model cannot embed Effect's longest 7,103-token AST chunk on the tested DML GPU even as a single-item batch. Do not silently truncate, re-chunk only one model, or mix CPU and GPU vectors to @@ -252,6 +258,14 @@ single-coordinate perturbations, epsilon-neighbor fraction, and median/worst hol currently deterministic with one seed and restart; those counts are explicit rather than implying unmeasured restart stability. +Schema 26 reports binary NDCG at 5, 10, 20, and 50 for raw query-form/repository/model rows and for +aggregate, fusion, and partitioned holdouts. The `direct` objective prioritizes NDCG@5, while the +`direct-recall-first` ablation retains the historical Recall@5-first priority under the same folds, +guardrails, and seeds. Both direct objectives retain Recall@20, Recall@50, and ContextRecall@4096 as +hard coverage guardrails. Reranker-top20 and reranker-top50 remain recall-first and retain the complete +Recall@5/10/20/50 plus ContextRecall@4096 guardrail set. The direct ablation selects from the same archive +without consuming another beam-diversity slot, preserving the original search budget. + Each router result also records a deterministic `random-scout` baseline using the same parameter grid and global-scout budget. `Random R@20` and `Random Ctx@4k` show whether the structured search beats that equal-budget random sample; the artifact records the effective seed and candidate count for reproduction. diff --git a/benchmarks/retrieval/evaluation/collect.ts b/benchmarks/retrieval/evaluation/collect.ts index 7090058..fbdae95 100644 --- a/benchmarks/retrieval/evaluation/collect.ts +++ b/benchmarks/retrieval/evaluation/collect.ts @@ -36,6 +36,7 @@ import { foldKey } from "./folds.js" import { contextRecallAtBudget, goldTargetRanks, + normalizedDiscountedCumulativeGain, recallAt, reciprocalRank, resolveGoldTargets, @@ -257,6 +258,10 @@ const buildModelSamples = ( recallAt10: recallAt(ranked, targets, 10), recallAt20: recallAt(ranked, targets, 20), recallAt50: recallAt(ranked, targets, 50), + ndcgAt5: normalizedDiscountedCumulativeGain(ranked, targets, 5), + ndcgAt10: normalizedDiscountedCumulativeGain(ranked, targets, 10), + ndcgAt20: normalizedDiscountedCumulativeGain(ranked, targets, 20), + ndcgAt50: normalizedDiscountedCumulativeGain(ranked, targets, 50), successAt10: successAt(ranked, targets, 10), successAt20: successAt(ranked, targets, 20), reciprocalRank: reciprocalRank(ranked, targets), diff --git a/benchmarks/retrieval/evaluation/metrics-core.d.mts b/benchmarks/retrieval/evaluation/metrics-core.d.mts new file mode 100644 index 0000000..e161c2c --- /dev/null +++ b/benchmarks/retrieval/evaluation/metrics-core.d.mts @@ -0,0 +1,6 @@ +/** Binary NDCG at K for ranked chunk IDs and resolved exact-target chunk IDs. */ +export declare const binaryNdcgAt: ( + ranked: readonly number[], + targets: readonly (readonly number[])[], + k: number, +) => number diff --git a/benchmarks/retrieval/evaluation/metrics-core.mjs b/benchmarks/retrieval/evaluation/metrics-core.mjs new file mode 100644 index 0000000..7511ad5 --- /dev/null +++ b/benchmarks/retrieval/evaluation/metrics-core.mjs @@ -0,0 +1,19 @@ +/** Binary NDCG at K for ranked chunk IDs and resolved exact-target chunk IDs. */ +export const binaryNdcgAt = (ranked, targets, k) => { + const relevant = new Set(targets.flatMap((target) => target)) + if (relevant.size === 0 || k <= 0) return 0 + + const seen = new Set() + let discountedGain = 0 + for (let rank = 0; rank < Math.min(k, ranked.length); rank++) { + const chunkIndex = ranked[rank] + if (seen.has(chunkIndex)) continue + seen.add(chunkIndex) + if (relevant.has(chunkIndex)) discountedGain += 1 / Math.log2(rank + 2) + } + + let idealDiscountedGain = 0 + for (let rank = 0; rank < Math.min(k, relevant.size); rank++) + idealDiscountedGain += 1 / Math.log2(rank + 2) + return discountedGain / idealDiscountedGain +} diff --git a/benchmarks/retrieval/evaluation/metrics.ts b/benchmarks/retrieval/evaluation/metrics.ts index 24f19aa..cdf2de2 100644 --- a/benchmarks/retrieval/evaluation/metrics.ts +++ b/benchmarks/retrieval/evaluation/metrics.ts @@ -1,5 +1,6 @@ import type { Chunk } from "../../../src/domain/chunk.js" import type { RankedChunk } from "../../../src/domain/ports.js" +import { binaryNdcgAt } from "./metrics-core.mjs" import type { GoldLocation } from "./types.js" /** Indexed identifiers retained per chunk so gold symbols can be matched exactly. */ @@ -63,6 +64,19 @@ export const reciprocalRank = ( return rank < 0 ? 0 : 1 / (rank + 1) } +/** Binary NDCG at K over unique chunks that resolve at least one exact gold target. */ +export const normalizedDiscountedCumulativeGain = ( + ranked: readonly RankedChunk[], + targets: readonly ReadonlySet[], + k: number, +): number => { + return binaryNdcgAt( + ranked.map((entry) => entry.chunkIndex), + targets.map((target) => [...target]), + k, + ) +} + /** One-based best rank for every authored target, or null when the channel did not retrieve it. */ export const goldTargetRanks = ( ranked: readonly RankedChunk[], diff --git a/benchmarks/retrieval/evaluation/optimization-profiles.ts b/benchmarks/retrieval/evaluation/optimization-profiles.ts index 3c46402..95df699 100644 --- a/benchmarks/retrieval/evaluation/optimization-profiles.ts +++ b/benchmarks/retrieval/evaluation/optimization-profiles.ts @@ -9,6 +9,10 @@ import { const NonNegativeNumber = Schema.Number.check(Schema.isGreaterThanOrEqualTo(0)) const QualityMetrics = [ + "ndcgAt5", + "ndcgAt10", + "ndcgAt20", + "ndcgAt50", "recallAt5", "recallAt10", "recallAt20", @@ -73,7 +77,10 @@ const objective = ( ): typeof MetricObjectiveSchema.Type => ({ name, priority, - guardrailMetrics: ["recallAt5", "recallAt10", "recallAt20", "recallAt50", "contextRecallAt4096"], + guardrailMetrics: + name === "direct" + ? ["recallAt20", "recallAt50", "contextRecallAt4096"] + : ["recallAt5", "recallAt10", "recallAt20", "recallAt50", "contextRecallAt4096"], guardrailTolerance: 0.01, contextBudget: 4_096, }) @@ -147,6 +154,9 @@ export const SEARCH_PRIORITY_PROFILE = profile( COMPATIBILITY_PROFILE_CONFIG, { identifier: 1, agentTask: 2, naturalQuestion: 3, searchPhrase: 4 }, objective("direct", [ + "ndcgAt5", + "ndcgAt10", + "ndcgAt20", "recallAt5", "recallAt10", "contextRecallAt4096", @@ -183,6 +193,9 @@ export const OPTIMIZATION_PROFILES = { BASIC_EXPLORATION_PROFILE_CONFIG, { identifier: 1, agentTask: 3, naturalQuestion: 4, searchPhrase: 2 }, objective("direct", [ + "ndcgAt5", + "ndcgAt10", + "ndcgAt20", "recallAt5", "recallAt10", "recallAt20", diff --git a/benchmarks/retrieval/evaluation/prepared-fusion-core.mjs b/benchmarks/retrieval/evaluation/prepared-fusion-core.mjs index d8a9f64..fdfd21c 100644 --- a/benchmarks/retrieval/evaluation/prepared-fusion-core.mjs +++ b/benchmarks/retrieval/evaluation/prepared-fusion-core.mjs @@ -1,3 +1,5 @@ +import { binaryNdcgAt } from "./metrics-core.mjs" + export const evaluatePreparedContributions = (matrix, weights) => { const activeMask = Number(weights.identity > 0) | @@ -64,6 +66,10 @@ export const evaluateCandidate = (snapshot, candidate) => { let recall10 = 0 let recall20 = 0 let recall50 = 0 + let ndcg5 = 0 + let ndcg10 = 0 + let ndcg20 = 0 + let ndcg50 = 0 let contextRecall = 0 let meanReciprocalRank = 0 let totalWeight = 0 @@ -81,6 +87,10 @@ export const evaluateCandidate = (snapshot, candidate) => { recall10 += weight * recallAt(ranked, sample.targets, 10) recall20 += weight * recallAt(ranked, sample.targets, 20) recall50 += weight * recallAt(ranked, sample.targets, 50) + ndcg5 += weight * binaryNdcgAt(ranked, sample.targets, 5) + ndcg10 += weight * binaryNdcgAt(ranked, sample.targets, 10) + ndcg20 += weight * binaryNdcgAt(ranked, sample.targets, 20) + ndcg50 += weight * binaryNdcgAt(ranked, sample.targets, 50) contextRecall += weight * contextRecallAtBudget(ranked, sample, 4096) meanReciprocalRank += weight * reciprocalRank(ranked, sample.targets) totalWeight += weight @@ -92,6 +102,10 @@ export const evaluateCandidate = (snapshot, candidate) => { recallAt10: 0, recallAt20: 0, recallAt50: 0, + ndcgAt5: 0, + ndcgAt10: 0, + ndcgAt20: 0, + ndcgAt50: 0, contextRecallAt4096: 0, meanReciprocalRank: 0, } @@ -101,6 +115,10 @@ export const evaluateCandidate = (snapshot, candidate) => { recallAt10: recall10 / totalWeight, recallAt20: recall20 / totalWeight, recallAt50: recall50 / totalWeight, + ndcgAt5: ndcg5 / totalWeight, + ndcgAt10: ndcg10 / totalWeight, + ndcgAt20: ndcg20 / totalWeight, + ndcgAt50: ndcg50 / totalWeight, contextRecallAt4096: contextRecall / totalWeight, meanReciprocalRank: meanReciprocalRank / totalWeight, } diff --git a/benchmarks/retrieval/evaluation/promotion-evidence.ts b/benchmarks/retrieval/evaluation/promotion-evidence.ts index 7a892bf..a5c065c 100644 --- a/benchmarks/retrieval/evaluation/promotion-evidence.ts +++ b/benchmarks/retrieval/evaluation/promotion-evidence.ts @@ -14,6 +14,10 @@ import type { const BOOTSTRAP_SAMPLES = 1_000 const STABILITY_EPSILON = 0.005 const QUALITY_METRICS: readonly QualityMetric[] = [ + "ndcgAt5", + "ndcgAt10", + "ndcgAt20", + "ndcgAt50", "recallAt5", "recallAt10", "recallAt20", @@ -124,7 +128,8 @@ const buildUncertainty = (rows: readonly PromotionHoldoutRow[]): readonly Holdou } const objectiveMetric = (objective: RouterObjective): QualityMetric => { - if (objective === "direct") return "recallAt5" + if (objective === "direct") return "ndcgAt5" + if (objective === "direct-recall-first") return "recallAt5" return objective === "reranker-top20" ? "recallAt20" : "recallAt50" } diff --git a/benchmarks/retrieval/evaluation/report.ts b/benchmarks/retrieval/evaluation/report.ts index 3285f96..11c5161 100644 --- a/benchmarks/retrieval/evaluation/report.ts +++ b/benchmarks/retrieval/evaluation/report.ts @@ -137,6 +137,8 @@ export const renderMarkdownReport = (artifact: BenchmarkArtifact): string => { .map(([kind, weight]) => `${kind}=${weight}`) .join(", ")}.`, "", + "The `direct` objective selects NDCG@5 first; `direct-recall-first` is the matched historical-objective ablation. Reranker objectives remain recall-first.", + "", `Validation protocol: candidates use ${artifact.validationProtocol.selection}; holdouts are ${artifact.validationProtocol.holdouts.join(" and ")}; final promotion requires untouched ${artifact.validationProtocol.finalTest.strategy} fold ${artifact.validationProtocol.finalTest.fold}.`, "", `Search strategy: \`${artifact.searchStrategy.algorithm}\` (${artifact.searchStrategy.globalScouts} global scouts, beam ${artifact.searchStrategy.beamWidth}, ${artifact.searchStrategy.coordinatePasses} coordinate passes, ${artifact.searchStrategy.proxySampleFraction * 100}% proxy with minimum ${artifact.searchStrategy.proxyMinimumSamples}, ${strategyFactorLabel} factor ${strategyFactor}x).`, @@ -162,14 +164,14 @@ export const renderMarkdownReport = (artifact: BenchmarkArtifact): string => { `| ${run.repository} | ${run.model} | ${run.tokenizerModel} | ${run.batchSize} | ${duration(run.chunkEmbeddingDurationMs)} | ${duration(run.queryTokenizationDurationMs)} |`, ), "", - "| Repository | Model | Query form | Variant | R@5 | R@10 | R@20 | R@50 | S@10 | S@20 | MRR | Ctx@2k | Ctx@4k |", - "| --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", + "| Repository | Model | Query form | Variant | R@5 | R@10 | R@20 | R@50 | NDCG@5 | NDCG@10 | NDCG@20 | NDCG@50 | S@10 | S@20 | MRR | Ctx@2k | Ctx@4k |", + "| --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", ] for (const [key, rows] of [...groups].sort(([left], [right]) => left.localeCompare(right))) { const [repository, model, queryKind, variant] = key.split("\0") lines.push( - `| ${repository} | ${model} | ${queryKind} | ${variant} | ${percent(average(rows, (row) => row.recallAt5))} | ${percent(average(rows, (row) => row.recallAt10))} | ${percent(average(rows, (row) => row.recallAt20))} | ${percent(average(rows, (row) => row.recallAt50))} | ${percent(average(rows, (row) => Number(row.successAt10)))} | ${percent(average(rows, (row) => Number(row.successAt20)))} | ${average(rows, (row) => row.reciprocalRank).toFixed(3)} | ${percent(average(rows, (row) => row.contextRecall["2048"] ?? 0))} | ${percent(average(rows, (row) => row.contextRecall["4096"] ?? 0))} |`, + `| ${repository} | ${model} | ${queryKind} | ${variant} | ${percent(average(rows, (row) => row.recallAt5))} | ${percent(average(rows, (row) => row.recallAt10))} | ${percent(average(rows, (row) => row.recallAt20))} | ${percent(average(rows, (row) => row.recallAt50))} | ${percent(average(rows, (row) => row.ndcgAt5))} | ${percent(average(rows, (row) => row.ndcgAt10))} | ${percent(average(rows, (row) => row.ndcgAt20))} | ${percent(average(rows, (row) => row.ndcgAt50))} | ${percent(average(rows, (row) => Number(row.successAt10)))} | ${percent(average(rows, (row) => Number(row.successAt20)))} | ${average(rows, (row) => row.reciprocalRank).toFixed(3)} | ${percent(average(rows, (row) => row.contextRecall["2048"] ?? 0))} | ${percent(average(rows, (row) => row.contextRecall["4096"] ?? 0))} |`, ) } @@ -297,7 +299,7 @@ export const renderMarkdownReport = (artifact: BenchmarkArtifact): string => { "", "## Evidence Router Holdouts", "", - "One shared search produces direct, reranker-top20, and reranker-top50 candidates. The current Production router is the guardrail baseline; static and dynamic validation columns use the same fusion method and excluded fold. A no eligible candidate result must not be promoted.", + "One shared search produces NDCG-first direct, Recall-first direct-ablation, reranker-top20, and reranker-top50 candidates. The current Production router is the guardrail baseline; static and dynamic validation columns use the same fusion method and excluded fold. A no eligible candidate result must not be promoted.", "", "| Model | Fusion | Objective | Strategy | Fold | Promotion | Params | Proxy evals | Full evals | Proxy agreement | Static I/C/B/D/S | Dynamic base I/C/B/D/S | Influence Score/Geometry/TermCoverage/PairwiseAgreement/DenseConfidence/Identifier/Length | Static R@5 | Dynamic R@5 | Static R@10 | Dynamic R@10 | Static R@20 | Dynamic R@20 | Dynamic R@50 | Static Ctx@4k | Dynamic Ctx@4k |", "| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", @@ -330,6 +332,22 @@ export const renderMarkdownReport = (artifact: BenchmarkArtifact): string => { ) } + lines.push( + "", + "## Evidence Router NDCG Holdouts", + "", + "These aggregate excluded-fold rows compare each fusion and objective with the current Production router under identical validation partitions.", + "", + "| Model | Fusion | Objective | Strategy | Production NDCG@5 | Dynamic NDCG@5 | Production NDCG@10 | Dynamic NDCG@10 | Production NDCG@20 | Dynamic NDCG@20 | Production NDCG@50 | Dynamic NDCG@50 |", + "| --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", + ) + for (const [key, rows] of routerGroups) { + const [model, fusion, objective, strategy] = key.split("\0") + lines.push( + `| ${model} | ${fusion} | ${objective} | ${strategy} | ${percent(weightedAverage(rows, (row) => row.productionValidation.ndcgAt5))} | ${percent(weightedAverage(rows, (row) => row.validation.ndcgAt5))} | ${percent(weightedAverage(rows, (row) => row.productionValidation.ndcgAt10))} | ${percent(weightedAverage(rows, (row) => row.validation.ndcgAt10))} | ${percent(weightedAverage(rows, (row) => row.productionValidation.ndcgAt20))} | ${percent(weightedAverage(rows, (row) => row.validation.ndcgAt20))} | ${percent(weightedAverage(rows, (row) => row.productionValidation.ndcgAt50))} | ${percent(weightedAverage(rows, (row) => row.validation.ndcgAt50))} |`, + ) + } + const holdoutRows = [ ...artifact.fusionSearch.flatMap((result) => result.holdoutBreakdown.map((holdout) => ({ @@ -368,6 +386,20 @@ export const renderMarkdownReport = (artifact: BenchmarkArtifact): string => { ) } + lines.push( + "", + "### Holdout NDCG Breakdown", + "", + "| Model | Fusion | Objective | Strategy | Fold | Partition | Candidate NDCG@5 | Baseline NDCG@5 | Candidate NDCG@10 | Baseline NDCG@10 | Candidate NDCG@20 | Baseline NDCG@20 | Candidate NDCG@50 | Baseline NDCG@50 |", + "| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", + ) + for (const row of holdoutRows) { + const holdout: HoldoutQuality = row.holdout + lines.push( + `| ${row.model} | ${row.fusion} | ${row.objective} | ${row.strategy} | ${row.fold} | ${holdout.dimension}:${holdout.name} | ${percent(holdout.candidate.ndcgAt5)} | ${percent(holdout.baseline.ndcgAt5)} | ${percent(holdout.candidate.ndcgAt10)} | ${percent(holdout.baseline.ndcgAt10)} | ${percent(holdout.candidate.ndcgAt20)} | ${percent(holdout.baseline.ndcgAt20)} | ${percent(holdout.candidate.ndcgAt50)} | ${percent(holdout.baseline.ndcgAt50)} |`, + ) + } + lines.push(...renderPromotionEvidence(artifact)) lines.push( diff --git a/benchmarks/retrieval/evaluation/types.ts b/benchmarks/retrieval/evaluation/types.ts index 513671e..d00c170 100644 --- a/benchmarks/retrieval/evaluation/types.ts +++ b/benchmarks/retrieval/evaluation/types.ts @@ -29,7 +29,7 @@ const BenchmarkQuestionSchema = Schema.Struct({ queries: QueryFormsSchema, category: Schema.String, difficulty: Schema.Literals(["easy", "medium", "hard"]), - groundTruth: Schema.Array(GoldLocationSchema), + groundTruth: Schema.NonEmptyArray(GoldLocationSchema), }) /** Versioned manifest for one pinned real-world benchmark repository. */ @@ -59,7 +59,12 @@ export type QueryKind = keyof typeof QueryFormsSchema.Type type FusionMethod = ProductionFusionMethod /** Product retrieval scenario used to choose a router candidate. */ -export const ROUTER_OBJECTIVES = ["direct", "reranker-top20", "reranker-top50"] as const +export const ROUTER_OBJECTIVES = [ + "direct", + "direct-recall-first", + "reranker-top20", + "reranker-top50", +] as const export type RouterObjective = (typeof ROUTER_OBJECTIVES)[number] /** Versioned evidence-router search strategies recorded in benchmark artifacts. */ @@ -145,6 +150,10 @@ export interface QueryMeasurement { readonly recallAt10: number readonly recallAt20: number readonly recallAt50: number + readonly ndcgAt5: number + readonly ndcgAt10: number + readonly ndcgAt20: number + readonly ndcgAt50: number readonly successAt10: boolean readonly successAt20: boolean readonly reciprocalRank: number @@ -168,6 +177,10 @@ export interface QualitySummary { readonly recallAt10: number readonly recallAt20: number readonly recallAt50: number + readonly ndcgAt5: number + readonly ndcgAt10: number + readonly ndcgAt20: number + readonly ndcgAt50: number readonly contextRecallAt4096: number readonly meanReciprocalRank: number } @@ -424,7 +437,7 @@ export interface BenchmarkTimings { /** Reproducible machine-readable output of one complete benchmark run. */ export interface BenchmarkArtifact { - readonly schemaVersion: 25 + readonly schemaVersion: 26 /** Profile controlling benchmark coverage without changing retrieval behavior. */ readonly benchmarkProfile: BenchmarkProfile /** Versioned objective profile used for candidate selection and aggregate metrics. */ diff --git a/benchmarks/retrieval/evaluation/weight-search.ts b/benchmarks/retrieval/evaluation/weight-search.ts index d6e0125..05595a8 100644 --- a/benchmarks/retrieval/evaluation/weight-search.ts +++ b/benchmarks/retrieval/evaluation/weight-search.ts @@ -28,7 +28,12 @@ import { type CandidateEvaluationQueue, type EvaluationCandidate, } from "../execution/candidate-evaluation-pool.js" -import { contextRecallAtBudget, recallAt, reciprocalRank } from "./metrics.js" +import { + contextRecallAtBudget, + normalizedDiscountedCumulativeGain, + recallAt, + reciprocalRank, +} from "./metrics.js" import { SEARCH_PRIORITY_PROFILE, type OptimizationProfile } from "./optimization-profiles.js" import { prepareFusion, type PreparedFusionEvaluator } from "./prepared-fusion.js" import { buildGuardrailBlockers } from "./promotion-evidence.js" @@ -177,6 +182,10 @@ const summarizeRanked = ( recallAt10: 0, recallAt20: 0, recallAt50: 0, + ndcgAt5: 0, + ndcgAt10: 0, + ndcgAt20: 0, + ndcgAt50: 0, contextRecallAt4096: 0, meanReciprocalRank: 0, } @@ -185,6 +194,10 @@ const summarizeRanked = ( let recall10 = 0 let recall20 = 0 let recall50 = 0 + let ndcg5 = 0 + let ndcg10 = 0 + let ndcg20 = 0 + let ndcg50 = 0 let contextRecall = 0 let mrr = 0 let totalWeight = 0 @@ -197,6 +210,10 @@ const summarizeRanked = ( recall10 += weight * recallAt(ranked, sampleTargets, 10) recall20 += weight * recallAt(ranked, sampleTargets, 20) recall50 += weight * recallAt(ranked, sampleTargets, 50) + ndcg5 += weight * normalizedDiscountedCumulativeGain(ranked, sampleTargets, 5) + ndcg10 += weight * normalizedDiscountedCumulativeGain(ranked, sampleTargets, 10) + ndcg20 += weight * normalizedDiscountedCumulativeGain(ranked, sampleTargets, 20) + ndcg50 += weight * normalizedDiscountedCumulativeGain(ranked, sampleTargets, 50) contextRecall += weight * contextRecallAtBudget(ranked, sampleTargets, chunks(sample), 4_096) mrr += weight * reciprocalRank(ranked, sampleTargets) totalWeight += weight @@ -207,6 +224,10 @@ const summarizeRanked = ( recallAt10: 0, recallAt20: 0, recallAt50: 0, + ndcgAt5: 0, + ndcgAt10: 0, + ndcgAt20: 0, + ndcgAt50: 0, contextRecallAt4096: 0, meanReciprocalRank: 0, } @@ -216,6 +237,10 @@ const summarizeRanked = ( recallAt10: recall10 / totalWeight, recallAt20: recall20 / totalWeight, recallAt50: recall50 / totalWeight, + ndcgAt5: ndcg5 / totalWeight, + ndcgAt10: ndcg10 / totalWeight, + ndcgAt20: ndcg20 / totalWeight, + ndcgAt50: ndcg50 / totalWeight, contextRecallAt4096: contextRecall / totalWeight, meanReciprocalRank: mrr / totalWeight, } @@ -297,6 +322,7 @@ const evidenceRouterGuardrailsMet = ( fusion: FusionMethod, baselines: GuardrailBaselines, profile: OptimizationProfile, + objective: RouterObjective, ): boolean => { const holdoutProfile = unweightedProfile(profile) const evaluate = (partition: GuardrailBaselinePartition): boolean => { @@ -306,13 +332,14 @@ const evidenceRouterGuardrailsMet = ( fusion, holdoutProfile, ) - return isWithinGuardrails(candidate, partition.baseline, holdoutProfile) + return isWithinGuardrails(candidate, partition.baseline, holdoutProfile, objective) } return ( isWithinGuardrails( summarizeEvidenceRouter(prepareEvidenceSamples(samples), config, fusion, profile), baselines.overall, profile, + objective, ) && baselines.partitions.every(evaluate) ) } @@ -366,6 +393,9 @@ const buildProxySamples = ( const OBJECTIVE_PRIORITIES: Readonly> = { direct: [ + "ndcgAt5", + "ndcgAt10", + "ndcgAt20", "recallAt5", "recallAt10", "contextRecallAt4096", @@ -373,6 +403,17 @@ const OBJECTIVE_PRIORITIES: Readonly> = { + direct: ["recallAt20", "recallAt50", "contextRecallAt4096"], + "direct-recall-first": ["recallAt20", "recallAt50", "contextRecallAt4096"], + "reranker-top20": ["recallAt5", "recallAt10", "recallAt20", "recallAt50", "contextRecallAt4096"], + "reranker-top50": ["recallAt5", "recallAt10", "recallAt20", "recallAt50", "contextRecallAt4096"], +} + +// The recall-first direct objective is an ablation over the same searched archive, not another +// source of beam candidates. This preserves the pre-ablation budget of two candidates per scenario. +const SEARCH_OBJECTIVES = ROUTER_OBJECTIVES.filter( + (objective): objective is Exclude => + objective !== "direct-recall-first", +) + const isWithinGuardrails = ( quality: QualitySummary, baseline: QualitySummary | undefined, profile: OptimizationProfile = SEARCH_PRIORITY_PROFILE, + objective?: RouterObjective, ): boolean => baseline === undefined || - profile.metricObjective.guardrailMetrics.every( + (objective === undefined + ? profile.metricObjective.guardrailMetrics + : OBJECTIVE_GUARDRAILS[objective] + ).every( (metric) => quality[metric] >= baseline[metric] - profile.metricObjective.guardrailTolerance, ) @@ -458,6 +517,7 @@ const buildHoldoutBreakdown = ( evaluationProfile: OptimizationProfile, ) => QualitySummary, profile: OptimizationProfile, + objective?: RouterObjective, ): readonly HoldoutQuality[] => { const holdoutProfile = unweightedProfile(profile) const baselines = buildGuardrailBaselines(samples, profile) @@ -473,7 +533,9 @@ const buildHoldoutBreakdown = ( partition.name, candidateQuality, partition.baseline, - evaluationProfile.metricObjective.guardrailMetrics, + objective === undefined + ? evaluationProfile.metricObjective.guardrailMetrics + : OBJECTIVE_GUARDRAILS[objective], evaluationProfile.metricObjective.guardrailTolerance, ) return { @@ -488,15 +550,16 @@ const buildHoldoutBreakdown = ( }) } -const compareObjectiveQuality = ( +/** Compare two quality summaries using one benchmark objective's deterministic priority. */ +export const compareObjectiveQuality = ( left: QualitySummary, right: QualitySummary, objective: RouterObjective, baseline?: QualitySummary, profile: OptimizationProfile = SEARCH_PRIORITY_PROFILE, ): number => { - const leftGuardrails = isWithinGuardrails(left, baseline, profile) - const rightGuardrails = isWithinGuardrails(right, baseline, profile) + const leftGuardrails = isWithinGuardrails(left, baseline, profile, objective) + const rightGuardrails = isWithinGuardrails(right, baseline, profile, objective) if (leftGuardrails !== rightGuardrails) return leftGuardrails ? -1 : 1 const priorities = profile.metricObjective.name === objective @@ -1140,8 +1203,8 @@ const selectObjectiveCandidates = ( profile: OptimizationProfile, ): readonly RouterCandidate[] => { const selected = new Map() - const perObjective = Math.max(1, Math.floor(limit / ROUTER_OBJECTIVES.length)) - for (const objective of ROUTER_OBJECTIVES) { + const perObjective = Math.max(1, Math.floor(limit / SEARCH_OBJECTIVES.length)) + for (const objective of SEARCH_OBJECTIVES) { const ranked = [...candidates] .sort((left, right) => compareRouterCandidates(left, right, objective, baseline, profile)) .slice(0, perObjective) @@ -1337,6 +1400,32 @@ export const selectEligibleCandidate = ( } } +/** Select one eligible archive candidate per objective using that objective's own comparator. */ +export const selectObjectiveArchiveCandidates = ( + candidates: readonly T[], + quality: (candidate: T) => QualitySummary, + isEligible: (candidate: T, objective: RouterObjective) => boolean, + baseline: QualitySummary, + profile: OptimizationProfile = SEARCH_PRIORITY_PROFILE, +): ReadonlyArray<{ + readonly objective: RouterObjective + readonly candidate: T | undefined + readonly promotionStatus: PromotionStatus +}> => + ROUTER_OBJECTIVES.map((objective) => { + const ranked = [...candidates].sort((left, right) => + compareObjectiveQuality(quality(left), quality(right), objective, baseline, profile), + ) + const selected = selectEligibleCandidate(ranked, (candidate) => + isEligible(candidate, objective), + ) + return { + objective, + candidate: selected.candidate ?? ranked[0], + promotionStatus: selected.promotionStatus, + } + }) + interface RouterSearchPreparation { readonly samples: readonly WeightSearchSample[] readonly evidenceSamples: readonly EvidenceSearchSample[] @@ -1504,43 +1593,30 @@ const selectBestEvidenceRouter = async ( const fallback = beam[0] if (fallback === undefined) throw new Error("Evidence router search produced no candidate") const candidates = [...searchContext.archive.values()] - const selections = useSuccessiveHalving - ? (() => { - const guardrailsMet = evidenceRouterGuardrailsMet( - samples, - fallback.config, - fusion, - guardrailBaselines, - profile, - ) - return ROUTER_OBJECTIVES.map((objective) => ({ - objective, - config: fallback.config, - quality: fallback.quality, - guardrailsMet, - promotionStatus: (guardrailsMet - ? "eligible" - : "no-eligible-candidate") as PromotionStatus, - })) - })() - : ROUTER_OBJECTIVES.map((objective) => { - const rankedCandidates = [...candidates].sort((left, right) => - compareRouterCandidates(left, right, objective, productionQuality, profile), - ) - const { candidate: eligibleCandidate, promotionStatus } = selectEligibleCandidate( - rankedCandidates, - (entry) => - evidenceRouterGuardrailsMet(samples, entry.config, fusion, guardrailBaselines, profile), - ) - const candidate = eligibleCandidate ?? rankedCandidates[0] ?? fallback - return { - objective, - config: candidate.config, - quality: candidate.quality, - guardrailsMet: eligibleCandidate !== undefined, - promotionStatus, - } - }) + const selections = selectObjectiveArchiveCandidates( + candidates, + ({ quality }) => quality, + (entry, objective) => + evidenceRouterGuardrailsMet( + samples, + entry.config, + fusion, + guardrailBaselines, + profile, + objective, + ), + productionQuality, + profile, + ).map(({ objective, candidate: selectedCandidate, promotionStatus }) => { + const candidate = selectedCandidate ?? fallback + return { + objective, + config: candidate.config, + quality: candidate.quality, + guardrailsMet: promotionStatus === "eligible", + promotionStatus, + } + }) const randomCandidate = randomSearch?.candidate ?? fallback return { selections, @@ -1918,6 +1994,7 @@ export const optimizeEvidenceRouter = async ( evaluationProfile, ), profile, + selection.objective, ) const guardrailsMet = holdoutBreakdown.every((holdout) => holdout.guardrailsMet) return { diff --git a/benchmarks/retrieval/runner.ts b/benchmarks/retrieval/runner.ts index 2b68326..c647108 100644 --- a/benchmarks/retrieval/runner.ts +++ b/benchmarks/retrieval/runner.ts @@ -210,7 +210,7 @@ export const runRetrievalBenchmark = ( ) const artifact: BenchmarkArtifact = { - schemaVersion: 25, + schemaVersion: 26, benchmarkProfile: profile, optimizationProfile, validationProtocol: { diff --git a/benchmarks/tests/channels.test.ts b/benchmarks/tests/channels.test.ts index 9173c44..9553043 100644 --- a/benchmarks/tests/channels.test.ts +++ b/benchmarks/tests/channels.test.ts @@ -12,6 +12,7 @@ import { buildIdentifierIndex } from "../../src/lib/retrieval/identifier-index.j import type { PreparedCorpus } from "../retrieval/corpus/prepare.js" import { contextRecallAtBudget, + normalizedDiscountedCumulativeGain, recallAt, reciprocalRank, resolveGoldTargets, @@ -22,11 +23,13 @@ import { rankLexicalChannels, RETRIEVAL_VARIANTS, } from "../retrieval/evaluation/ranking.js" -import { ROUTER_OBJECTIVES } from "../retrieval/evaluation/types.js" +import { ROUTER_OBJECTIVES, type QualitySummary } from "../retrieval/evaluation/types.js" import { + compareObjectiveQuality, optimizeEvidenceRouter, optimizeWeights, selectEligibleCandidate, + selectObjectiveArchiveCandidates, } from "../retrieval/evaluation/weight-search.js" const texts = [ @@ -366,6 +369,22 @@ describe("retrieval benchmark fixture", () => { expect(recallAt(ranked, targets, 1)).toBe(1) }) + it("measures binary NDCG without inflating duplicate or overlapping gold chunks", () => { + const ranked = [ + { chunkIndex: 2, score: 4 }, + { chunkIndex: 0, score: 3 }, + { chunkIndex: 0, score: 2 }, + { chunkIndex: 1, score: 1 }, + ] + const targets = [new Set([0]), new Set([0, 1])] + + expect(normalizedDiscountedCumulativeGain(ranked, targets, 4)).toBeCloseTo( + (1 / Math.log2(3) + 1 / Math.log2(5)) / (1 + 1 / Math.log2(3)), + ) + expect(normalizedDiscountedCumulativeGain(ranked, [], 5)).toBe(0) + expect(normalizedDiscountedCumulativeGain(ranked, [new Set()], 5)).toBe(0) + }) + it("keeps equal-weight RRF separate from production routing", () => { const rankings = { identity: [ @@ -397,6 +416,10 @@ describe("retrieval benchmark fixture", () => { recallAt10: recallAt(ranked, sample.targets, 10), recallAt20: recallAt(ranked, sample.targets, 20), recallAt50: recallAt(ranked, sample.targets, 50), + ndcgAt5: normalizedDiscountedCumulativeGain(ranked, sample.targets, 5), + ndcgAt10: normalizedDiscountedCumulativeGain(ranked, sample.targets, 10), + ndcgAt20: normalizedDiscountedCumulativeGain(ranked, sample.targets, 20), + ndcgAt50: normalizedDiscountedCumulativeGain(ranked, sample.targets, 50), contextRecallAt4096: contextRecallAtBudget(ranked, sample.targets, sample.chunks, 4_096), meanReciprocalRank: reciprocalRank(ranked, sample.targets), }) @@ -542,6 +565,45 @@ describe("retrieval benchmark fixture", () => { expect(result.shapleyRecallAt20.identity).toBe(1) }) + it("compares NDCG-first direct selection with the matched Recall-first ablation", () => { + const quality = (ndcgAt5: number, recallAt5: number): QualitySummary => ({ + ndcgAt5, + ndcgAt10: ndcgAt5, + ndcgAt20: ndcgAt5, + ndcgAt50: ndcgAt5, + recallAt5, + recallAt10: recallAt5, + recallAt20: 1, + recallAt50: 1, + contextRecallAt4096: 1, + meanReciprocalRank: recallAt5, + }) + const ndcgFirst = quality(0.9, 0.7) + const recallFirst = quality(0.8, 0.8) + + expect(compareObjectiveQuality(ndcgFirst, recallFirst, "direct", recallFirst)).toBeLessThan(0) + expect(compareObjectiveQuality(ndcgFirst, recallFirst, "direct-recall-first")).toBeGreaterThan( + 0, + ) + expect( + compareObjectiveQuality(ndcgFirst, recallFirst, "reranker-top20", recallFirst), + ).toBeGreaterThan(0) + expect(compareObjectiveQuality(quality(0.9, 0.8), quality(0.1, 0.8), "reranker-top20")).toBe(0) + const selections = selectObjectiveArchiveCandidates( + [ + { id: "ndcg", quality: ndcgFirst }, + { id: "recall", quality: recallFirst }, + ], + ({ quality }) => quality, + () => true, + recallFirst, + ) + expect(selections.find(({ objective }) => objective === "direct")?.candidate?.id).toBe("ndcg") + expect( + selections.find(({ objective }) => objective === "direct-recall-first")?.candidate?.id, + ).toBe("recall") + }) + it("selects one evidence router across queries with different reliable channels", async () => { const samples = makeEvidenceRouterSamples() diff --git a/benchmarks/tests/corpus.test.ts b/benchmarks/tests/corpus.test.ts index c164b7e..ba77cce 100644 --- a/benchmarks/tests/corpus.test.ts +++ b/benchmarks/tests/corpus.test.ts @@ -1,10 +1,11 @@ import { expect, it } from "@effect/vitest" -import { Effect } from "effect" +import { Effect, Schema } from "effect" import type { ChunkingOptions } from "../../src/domain/ports.js" import { prepareCorpus } from "../retrieval/corpus/prepare.js" import { loadCorpusManifests, prepareRepository } from "../retrieval/corpus/repository.js" import { resolveGoldTargets } from "../retrieval/evaluation/metrics.js" +import { CorpusManifestSchema } from "../retrieval/evaluation/types.js" const validationChunkingOptions: ChunkingOptions = { maxTokens: Number.MAX_SAFE_INTEGER, @@ -13,6 +14,36 @@ const validationChunkingOptions: ChunkingOptions = { onDiagnostic: () => Effect.void, } +it("rejects benchmark questions without exact ground truth", () => { + expect(() => + Schema.decodeUnknownSync(CorpusManifestSchema)({ + schemaVersion: 2, + id: "fixture", + repository: "owner/repository", + revision: "abc123", + language: "TypeScript", + size: "small", + includeRoots: ["src"], + excludePaths: [], + extensions: [".ts"], + questions: [ + { + id: "missing-ground-truth", + queries: { + identifier: "target", + searchPhrase: "target", + naturalQuestion: "Where is target?", + agentTask: "Find target", + }, + category: "navigation", + difficulty: "easy", + groundTruth: [], + }, + ], + }), + ).toThrow() +}) + // This validation intentionally reads real pinned checkouts; memfs is used by other adapter tests. it.effect("resolves every authored gold symbol in each pinned corpus", () => Effect.gen(function* () { diff --git a/benchmarks/tests/optimization-profiles.test.ts b/benchmarks/tests/optimization-profiles.test.ts index 34d5cb5..086360e 100644 --- a/benchmarks/tests/optimization-profiles.test.ts +++ b/benchmarks/tests/optimization-profiles.test.ts @@ -15,6 +15,12 @@ it("records the authored weighted search-priority objective without a profile sc }) expect("schemaVersion" in SEARCH_PRIORITY_PROFILE).toBe(false) expect(SEARCH_PRIORITY_PROFILE.provenance).toBe("authored-seed") + expect(SEARCH_PRIORITY_PROFILE.metricObjective.priority[0]).toBe("ndcgAt5") + expect(SEARCH_PRIORITY_PROFILE.metricObjective.guardrailMetrics).toEqual([ + "recallAt20", + "recallAt50", + "contextRecallAt4096", + ]) expect(SEARCH_PRIORITY_PROFILE.fusionConfig.baseWeights).toEqual({ identity: 3, camelcase: 1.5, @@ -35,6 +41,12 @@ it("records the authored weighted search-priority objective without a profile sc expect( OPTIMIZATION_PROFILES["code-navigation"].fusionConfig.baseWeights.identity, ).toBeGreaterThan(OPTIMIZATION_PROFILES["natural-language"].fusionConfig.baseWeights.identity) + expect(OPTIMIZATION_PROFILES["code-navigation"].metricObjective.priority).not.toContain( + "ndcgAt20", + ) + expect(OPTIMIZATION_PROFILES["natural-language"].metricObjective.priority).not.toContain( + "ndcgAt50", + ) }) it("rejects semantically empty benchmark profiles", () => { diff --git a/benchmarks/tests/promotion-evidence.test.ts b/benchmarks/tests/promotion-evidence.test.ts index d496739..1714492 100644 --- a/benchmarks/tests/promotion-evidence.test.ts +++ b/benchmarks/tests/promotion-evidence.test.ts @@ -8,13 +8,18 @@ import { import type { PromotionHoldoutRow } from "../retrieval/evaluation/promotion-evidence.js" import type { HoldoutQuality, QualitySummary } from "../retrieval/evaluation/types.js" -const quality = (recallAt20: number): QualitySummary => ({ +const quality = (recallAt20: number, overrides: Partial = {}): QualitySummary => ({ + ndcgAt5: recallAt20, + ndcgAt10: recallAt20, + ndcgAt20: recallAt20, + ndcgAt50: recallAt20, recallAt5: recallAt20, recallAt10: recallAt20, recallAt20, recallAt50: recallAt20, contextRecallAt4096: recallAt20, meanReciprocalRank: recallAt20, + ...overrides, }) const holdout = (candidate: number, baseline: number): HoldoutQuality => ({ @@ -131,4 +136,31 @@ describe("promotion evidence", () => { restarts: 1, }) }) + + it("uses each direct objective's primary metric for stability diagnostics", () => { + const row = (objective: "direct" | "direct-recall-first"): PromotionHoldoutRow => ({ + model: "fixture", + fusion: "dbsf", + objective, + strategy: "grouped-5-fold", + fold: "1", + validation: quality(1, { ndcgAt5: 0.9, recallAt5: 0.1 }), + productionValidation: quality(1, { ndcgAt5: 0.8, recallAt5: 0.9 }), + holdoutBreakdown: [holdout(1, 1)], + config: PRODUCTION_COMPATIBILITY_CONFIG, + }) + const evidence = derivePromotionEvidence( + [row("direct"), row("direct-recall-first")], + ["grouped-5-fold"], + { strategy: "grouped-5-fold", fold: "1" }, + ) + + expect(evidence.find(({ objective }) => objective === "direct")?.stability).toMatchObject({ + medianHoldoutDrop: -0.1, + worstCaseHoldoutDrop: 0, + }) + expect( + evidence.find(({ objective }) => objective === "direct-recall-first")?.stability, + ).toMatchObject({ medianHoldoutDrop: 0.8, worstCaseHoldoutDrop: 0.8 }) + }) }) diff --git a/benchmarks/tests/report.test.ts b/benchmarks/tests/report.test.ts new file mode 100644 index 0000000..6f89143 --- /dev/null +++ b/benchmarks/tests/report.test.ts @@ -0,0 +1,86 @@ +import { expect, it } from "@effect/vitest" + +import { SEARCH_PRIORITY_PROFILE } from "../retrieval/evaluation/optimization-profiles.js" +import { renderMarkdownReport } from "../retrieval/evaluation/report.js" +import { ROUTER_SEARCH_STRATEGIES, type BenchmarkArtifact } from "../retrieval/evaluation/types.js" + +const artifact = { + schemaVersion: 26, + benchmarkProfile: "smoke", + optimizationProfile: SEARCH_PRIORITY_PROFILE, + validationProtocol: { + selection: "development-only", + holdouts: ["grouped-5-fold"], + finalTest: { kind: "untouched-grouped-fold", strategy: "grouped-5-fold", fold: "5" }, + }, + generatedAt: "2026-08-06T00:00:00.000Z", + searchStrategy: ROUTER_SEARCH_STRATEGIES["proxy-promotion"], + timings: { + totalDurationMs: 0, + corpusPreparationDurationMs: 0, + embeddingDurationMs: 0, + retrievalDurationMs: 0, + weightSearchDurationMs: 0, + fusionSearchDurationMs: 0, + evidenceRouterSearchDurationMs: 0, + candidateQueueStartupDurationMs: 0, + candidateQueueShutdownDurationMs: 0, + }, + chunkConfig: { chunkTokens: 512, overlapLines: 0 }, + contextTokenEstimator: "utf8-bytes-divided-by-four", + contextBudgets: [2_048, 4_096], + models: ["fixture-model"], + repositories: [], + evaluationCases: [], + embeddingRuns: [], + sparseEmbeddingRuns: [], + measurements: [ + { + repository: "fixture", + language: "TypeScript", + size: "small", + revision: "abc123", + model: "fixture-model", + variant: "rrf", + questionId: "fixture-1", + queryKind: "agentTask", + query: "find target", + category: "navigation", + difficulty: "easy", + groupedFold: 1, + recallAt5: 1, + recallAt10: 1, + recallAt20: 1, + recallAt50: 1, + ndcgAt5: 0.5, + ndcgAt10: 0.6, + ndcgAt20: 0.7, + ndcgAt50: 0.8, + successAt10: true, + successAt20: true, + reciprocalRank: 0.5, + goldRanks: [2], + contextRecall: { "2048": 1, "4096": 1 }, + queryDurationMs: 0, + }, + ], + weightSearch: [], + recommendedWeights: [], + productionRouterSearch: [], + fusionSearch: [], + recommendedFusionWeights: [], + evidenceRouterSearch: [], + recommendedEvidenceRouters: [], + promotionEvidence: [], +} satisfies BenchmarkArtifact + +it("renders the complete NDCG and direct-objective contract", () => { + const report = renderMarkdownReport(artifact) + + expect(report).toContain("NDCG@5 | NDCG@10 | NDCG@20 | NDCG@50") + expect(report).toContain("| 50.0% | 60.0% | 70.0% | 80.0% |") + expect(report).toContain("`direct` objective selects NDCG@5 first") + expect(report).toContain("Recall-first direct-ablation") + expect(report).toContain("reranker-top20") + expect(report).toContain("reranker-top50") +}) diff --git a/benchmarks/tests/retrieval.test.ts b/benchmarks/tests/retrieval.test.ts index 44e8433..841f618 100644 --- a/benchmarks/tests/retrieval.test.ts +++ b/benchmarks/tests/retrieval.test.ts @@ -20,7 +20,7 @@ const foldQuestions = (prefix: string) => }, category: ["architecture", "routing", "symbol-lookup"][Math.floor(index / 4)], difficulty: ["easy", "medium", "hard"][index % 3] as "easy" | "medium" | "hard", - groundTruth: [], + groundTruth: [{ file: "src/fixture.ts", symbol: `target${index + 1}` }] as const, })) type TestQuestion = ReturnType[number] @@ -103,7 +103,7 @@ const runProfile = (profile: BenchmarkProfile, groupedFolds: number, fusionMetho expect(artifact.evaluationCases.every(({ groundTruth }) => groundTruth.length > 0)).toBe(true) expect(artifact.models.length).toBeGreaterThan(0) expect(artifact.measurements.length).toBeGreaterThan(0) - expect(artifact.schemaVersion).toBe(25) + expect(artifact.schemaVersion).toBe(26) expect(artifact.searchStrategy).toEqual( ROUTER_SEARCH_STRATEGIES[resolveRouterSearchStrategy(process.env.PIX_BENCH_ROUTER_STRATEGY)], ) @@ -127,6 +127,9 @@ const runProfile = (profile: BenchmarkProfile, groupedFolds: number, fusionMetho expect(artifact.evidenceRouterSearch.length).toBe( artifact.models.length * routerFusionMethods * ROUTER_OBJECTIVES.length * holdoutsPerModel, ) + expect(new Set(artifact.evidenceRouterSearch.map(({ objective }) => objective))).toEqual( + new Set(ROUTER_OBJECTIVES), + ) expect(new Set(artifact.evidenceRouterSearch.map((row) => row.fusion)).size).toBe( routerFusionMethods, ) @@ -145,6 +148,19 @@ const runProfile = (profile: BenchmarkProfile, groupedFolds: number, fusionMetho expect(artifact.measurements.every((row) => row.recallAt20 >= row.recallAt10)).toBe(true) expect(artifact.measurements.every((row) => row.recallAt10 >= row.recallAt5)).toBe(true) expect(artifact.measurements.every((row) => row.recallAt50 >= row.recallAt20)).toBe(true) + expect( + artifact.measurements.every( + (row) => + row.ndcgAt5 >= 0 && + row.ndcgAt5 <= 1 && + row.ndcgAt10 >= 0 && + row.ndcgAt10 <= 1 && + row.ndcgAt20 >= 0 && + row.ndcgAt20 <= 1 && + row.ndcgAt50 >= 0 && + row.ndcgAt50 <= 1, + ), + ).toBe(true) expect(outputPath).toMatch(/benchmarks[\\/]results[\\/]retrieval-.*\.json$/) }) diff --git a/benchmarks/tests/worker-pool.test.ts b/benchmarks/tests/worker-pool.test.ts index 9bfbd7a..552e6b0 100644 --- a/benchmarks/tests/worker-pool.test.ts +++ b/benchmarks/tests/worker-pool.test.ts @@ -5,6 +5,7 @@ import { describe, expect, it } from "@effect/vitest" import type { Chunk } from "../../src/domain/chunk.js" import { SEARCH_PRIORITY_PROFILE } from "../retrieval/evaluation/optimization-profiles.js" import { prepareFusion } from "../retrieval/evaluation/prepared-fusion.js" +import { ROUTER_OBJECTIVES } from "../retrieval/evaluation/types.js" import { fitRecommendedEvidenceRouter, fitRecommendedFusionWeights, @@ -342,7 +343,7 @@ describe("benchmark candidate evaluation pool", () => { it("retains the serial evidence-router fit path", async () => { const result = await fitRecommendedEvidenceRouter("fixture", "dbsf", [searchSample]) - expect(result).toHaveLength(3) + expect(result).toHaveLength(ROUTER_OBJECTIVES.length) expect(result.every((candidate) => candidate.fitQuality.recallAt20 >= 0)).toBe(true) })