feat: add NDCG benchmark metrics

CONTEXT.md

@@ -297,6 +297,15 @@ must all be present and pass aggregate, query-form, and repository guardrails. F
 diagnostic and inherit that decision. The last grouped fold is the explicit untouched final test and
 must pass independently. Artifacts retain exact blocker values, deterministic grouped bootstrap
 intervals, and fold-level selection/local-perturbation stability diagnostics.
+Schema 26 adds binary NDCG@5/10/20/50 without changing the corpus or Production retrieval. Unique chunks
+matching at least one resolved exact file-plus-symbol target receive gain one; duplicates and chunks
+matching multiple targets cannot add repeated gain, and zero ideal gain returns NDCG zero. The direct
+objective selects NDCG@5 first and is compared with a `direct-recall-first` ablation under identical
+folds, seeds, and guardrails. Recall@20, Recall@50, and ContextRecall@4096 remain hard direct-retrieval
+coverage guardrails; reranker objectives remain recall-first with the full Recall@5/10/20/50 and
+ContextRecall@4096 guardrail set. Empty authored ground truth is rejected by the manifest schema. The
+Recall-first direct ablation selects from the same archive without changing the three-scenario beam
+budget, and both proxy-promotion and successive-halving perform objective-specific archive selection.
 
 ### Scorer
 

benchmarks/BASELINE.md

@@ -1,7 +1,26 @@
 # Preliminary Retrieval Baseline
 
 The schema-17 entries below are historical artifacts from the benchmark-owned Sparse implementation.
-Current schema-24 runs use the production SparseEmbedder and IndexStore without benchmark vector caches.
+Current schema-26 runs use the production SparseEmbedder and IndexStore without benchmark vector caches.
+
+## Schema 26: Binary NDCG Direct-Objective Smoke
+
+The `fd` MiniLM smoke artifact `retrieval-2026-08-06T18-27-13.604Z.json` compares the new NDCG@5-first
+`direct` objective with the historical `direct-recall-first` priority under the same grouped folds,
+seeds, DBSF fusion, and hard Recall@20/50 plus Context@4k guardrails. The artifact is local and ignored.
+
+| Objective      | NDCG@5 | NDCG@10 | NDCG@20 | NDCG@50 |   R@5 |  R@20 | Context@4k |
+| -------------- | -----: | ------: | ------: | ------: | ----: | ----: | ---------: |
+| NDCG@5-first   |  57.0% |   61.5% |   62.5% |   64.2% | 73.3% | 91.3% |      87.3% |
+| Recall@5-first |  59.8% |   63.2% |   64.2% |   65.8% | 76.7% | 91.3% |      87.3% |
+
+This single-repository smoke run validates the complete measurement and ablation path, but does not
+support promoting the NDCG-first candidate: Recall-first is stronger on every reported holdout NDCG
+cutoff and R@5 while tying the coverage guardrails. A full multi-repository run is still required for
+any Production decision; issue #172 itself remains benchmark-only.
+The matched Successive-Halving smoke artifact `retrieval-2026-08-06T18-27-50.835Z.json` also completed
+with objective-specific archive selection, validating that the ablation is executed rather than emitted
+as repeated labels in both supported search strategies.
 
 ## Schema 24: Selectable Router Search
 

benchmarks/README.md

@@ -9,7 +9,7 @@ surface.
 The suite answers five questions independently:
 
 1. Do identity, CamelCase, BM25, dense, and sparse retrieval each contribute useful candidates?
-2. Does production-weighted RRF improve `Recall@K` over individual channels and combinations?
+2. Does production-weighted RRF improve `Recall@K` and binary `NDCG@K` over individual channels and combinations?
 3. Does full RRF reduce the quality gap between a small general embedder and a code-specific one?
 4. How much authored ground truth fits into fixed context budgets?
 5. Can one evidence-based router outperform static weights without knowing the authored query form?
@@ -53,6 +53,12 @@ representations remain in the same validation fold. Gold targets are exact `file
 Matching uses extracted identifiers or exact declaration syntax inside the specified file; loose
 symbol-name segment matching is not accepted.
 
+The existing corpus needs no NDCG-specific labels. Every resolved chunk matching one or more exact
+gold targets has binary gain `1`; non-matching chunks have gain `0`. A chunk that resolves multiple
+targets and a duplicate occurrence of the same chunk each contribute at most once. NDCG is `0` when
+no gold target resolves. Manifest decoding rejects empty ground-truth arrays, and corpus preparation
+rejects authored targets that do not resolve to a chunk.
+
 | Corpus    | Revision                                   | Language   | Size band | Indexed scope                 |
 | --------- | ------------------------------------------ | ---------- | --------- | ----------------------------- |
 | FastAPI   | `95f8322ee1dcda7ceace7b1c4f6c9915b36d748f` | Python     | medium    | `fastapi/**/*.py`             |
@@ -95,7 +101,7 @@ vp run bench:retrieval:full
 
 `bench:retrieval` aliases `bench:retrieval:validate`. Every profile measures the same physical
 rankings and retrieval variants; profiles only control matrix size, holdout coverage, and expensive
-diagnostics. The selected profile is recorded in schema-24 artifacts without changing retrieval
+diagnostics. The selected profile is recorded in schema-26 artifacts without changing retrieval
 semantics. The full profile includes all three fusion methods; short profiles intentionally omit RRF
 to keep development runs fast.
 
@@ -158,8 +164,8 @@ defaulting to MiniLM. Select another with `PIX_BENCH_MODELS`. Supported values a
 The router search defaults to `proxy-promotion`. Set `PIX_BENCH_ROUTER_STRATEGY` to
 `successive-halving` to select the historical Successive-Halving variant. It uses the original
 lexicographic `R@20`, `R@10`, `Context@4k`, and MRR comparator plus its `halvingKeepFactor`.
-Both strategies use the same candidate evaluator and native worker queue, so their artifacts can be
-compared directly.
+Both strategies use the same candidate evaluator and native worker queue. Their final archive selection
+is objective-specific, so Direct and Reranker rows are real comparisons rather than repeated labels.
 
 The Jina code model cannot embed Effect's longest 7,103-token AST chunk on the tested DML GPU even as
 a single-item batch. Do not silently truncate, re-chunk only one model, or mix CPU and GPU vectors to
@@ -252,6 +258,14 @@ single-coordinate perturbations, epsilon-neighbor fraction, and median/worst hol
 currently deterministic with one seed and restart; those counts are explicit rather than implying
 unmeasured restart stability.
 
+Schema 26 reports binary NDCG at 5, 10, 20, and 50 for raw query-form/repository/model rows and for
+aggregate, fusion, and partitioned holdouts. The `direct` objective prioritizes NDCG@5, while the
+`direct-recall-first` ablation retains the historical Recall@5-first priority under the same folds,
+guardrails, and seeds. Both direct objectives retain Recall@20, Recall@50, and ContextRecall@4096 as
+hard coverage guardrails. Reranker-top20 and reranker-top50 remain recall-first and retain the complete
+Recall@5/10/20/50 plus ContextRecall@4096 guardrail set. The direct ablation selects from the same archive
+without consuming another beam-diversity slot, preserving the original search budget.
+
 Each router result also records a deterministic `random-scout` baseline using the same parameter grid and
 global-scout budget. `Random R@20` and `Random Ctx@4k` show whether the structured search beats that
 equal-budget random sample; the artifact records the effective seed and candidate count for reproduction.

benchmarks/retrieval/evaluation/collect.ts

@@ -36,6 +36,7 @@ import { foldKey } from "./folds.js"
 import {
   contextRecallAtBudget,
   goldTargetRanks,
+  normalizedDiscountedCumulativeGain,
   recallAt,
   reciprocalRank,
   resolveGoldTargets,
@@ -257,6 +258,10 @@ const buildModelSamples = (
           recallAt10: recallAt(ranked, targets, 10),
           recallAt20: recallAt(ranked, targets, 20),
           recallAt50: recallAt(ranked, targets, 50),
+          ndcgAt5: normalizedDiscountedCumulativeGain(ranked, targets, 5),
+          ndcgAt10: normalizedDiscountedCumulativeGain(ranked, targets, 10),
+          ndcgAt20: normalizedDiscountedCumulativeGain(ranked, targets, 20),
+          ndcgAt50: normalizedDiscountedCumulativeGain(ranked, targets, 50),
           successAt10: successAt(ranked, targets, 10),
           successAt20: successAt(ranked, targets, 20),
           reciprocalRank: reciprocalRank(ranked, targets),

benchmarks/retrieval/evaluation/metrics-core.d.mts

@@ -0,0 +1,6 @@
+/** Binary NDCG at K for ranked chunk IDs and resolved exact-target chunk IDs. */
+export declare const binaryNdcgAt: (
+  ranked: readonly number[],
+  targets: readonly (readonly number[])[],
+  k: number,
+) => number

benchmarks/retrieval/evaluation/metrics-core.mjs

@@ -0,0 +1,19 @@
+/** Binary NDCG at K for ranked chunk IDs and resolved exact-target chunk IDs. */
+export const binaryNdcgAt = (ranked, targets, k) => {
+  const relevant = new Set(targets.flatMap((target) => target))
+  if (relevant.size === 0 || k <= 0) return 0
+
+  const seen = new Set()
+  let discountedGain = 0
+  for (let rank = 0; rank < Math.min(k, ranked.length); rank++) {
+    const chunkIndex = ranked[rank]
+    if (seen.has(chunkIndex)) continue
+    seen.add(chunkIndex)
+    if (relevant.has(chunkIndex)) discountedGain += 1 / Math.log2(rank + 2)
+  }
+
+  let idealDiscountedGain = 0
+  for (let rank = 0; rank < Math.min(k, relevant.size); rank++)
+    idealDiscountedGain += 1 / Math.log2(rank + 2)
+  return discountedGain / idealDiscountedGain
+}

benchmarks/retrieval/evaluation/metrics.ts

@@ -1,5 +1,6 @@
 import type { Chunk } from "../../../src/domain/chunk.js"
 import type { RankedChunk } from "../../../src/domain/ports.js"
+import { binaryNdcgAt } from "./metrics-core.mjs"
 import type { GoldLocation } from "./types.js"
 
 /** Indexed identifiers retained per chunk so gold symbols can be matched exactly. */
@@ -63,6 +64,19 @@ export const reciprocalRank = (
   return rank < 0 ? 0 : 1 / (rank + 1)
 }
 
+/** Binary NDCG at K over unique chunks that resolve at least one exact gold target. */
+export const normalizedDiscountedCumulativeGain = (
+  ranked: readonly RankedChunk[],
+  targets: readonly ReadonlySet<number>[],
+  k: number,
+): number => {
+  return binaryNdcgAt(
+    ranked.map((entry) => entry.chunkIndex),
+    targets.map((target) => [...target]),
+    k,
+  )
+}
+
 /** One-based best rank for every authored target, or null when the channel did not retrieve it. */
 export const goldTargetRanks = (
   ranked: readonly RankedChunk[],

benchmarks/retrieval/evaluation/optimization-profiles.ts

@@ -9,6 +9,10 @@ import {
 
 const NonNegativeNumber = Schema.Number.check(Schema.isGreaterThanOrEqualTo(0))
 const QualityMetrics = [
+  "ndcgAt5",
+  "ndcgAt10",
+  "ndcgAt20",
+  "ndcgAt50",
   "recallAt5",
   "recallAt10",
   "recallAt20",
@@ -73,7 +77,10 @@ const objective = (
 ): typeof MetricObjectiveSchema.Type => ({
   name,
   priority,
-  guardrailMetrics: ["recallAt5", "recallAt10", "recallAt20", "recallAt50", "contextRecallAt4096"],
+  guardrailMetrics:
+    name === "direct"
+      ? ["recallAt20", "recallAt50", "contextRecallAt4096"]
+      : ["recallAt5", "recallAt10", "recallAt20", "recallAt50", "contextRecallAt4096"],
   guardrailTolerance: 0.01,
   contextBudget: 4_096,
 })
@@ -147,6 +154,9 @@ export const SEARCH_PRIORITY_PROFILE = profile(
   COMPATIBILITY_PROFILE_CONFIG,
   { identifier: 1, agentTask: 2, naturalQuestion: 3, searchPhrase: 4 },
   objective("direct", [
+    "ndcgAt5",
+    "ndcgAt10",
+    "ndcgAt20",
     "recallAt5",
     "recallAt10",
     "contextRecallAt4096",
@@ -176,13 +186,17 @@ export const OPTIMIZATION_PROFILES = {
       "contextRecallAt4096",
       "recallAt50",
       "meanReciprocalRank",
+      "ndcgAt20",
     ]),
   ),
   "basic-exploration": profile(
     "basic-exploration",
     BASIC_EXPLORATION_PROFILE_CONFIG,
     { identifier: 1, agentTask: 3, naturalQuestion: 4, searchPhrase: 2 },
     objective("direct", [
+      "ndcgAt5",
+      "ndcgAt10",
+      "ndcgAt20",
       "recallAt5",
       "recallAt10",
       "recallAt20",
@@ -202,6 +216,7 @@ export const OPTIMIZATION_PROFILES = {
       "recallAt5",
       "contextRecallAt4096",
       "meanReciprocalRank",
+      "ndcgAt50",
     ]),
   ),
 } as const

benchmarks/retrieval/evaluation/prepared-fusion-core.mjs

@@ -1,3 +1,5 @@
+import { binaryNdcgAt } from "./metrics-core.mjs"
+
 export const evaluatePreparedContributions = (matrix, weights) => {
   const activeMask =
     Number(weights.identity > 0) |
@@ -64,6 +66,10 @@ export const evaluateCandidate = (snapshot, candidate) => {
   let recall10 = 0
   let recall20 = 0
   let recall50 = 0
+  let ndcg5 = 0
+  let ndcg10 = 0
+  let ndcg20 = 0
+  let ndcg50 = 0
   let contextRecall = 0
   let meanReciprocalRank = 0
   let totalWeight = 0
@@ -81,6 +87,10 @@ export const evaluateCandidate = (snapshot, candidate) => {
     recall10 += weight * recallAt(ranked, sample.targets, 10)
     recall20 += weight * recallAt(ranked, sample.targets, 20)
     recall50 += weight * recallAt(ranked, sample.targets, 50)
+    ndcg5 += weight * binaryNdcgAt(ranked, sample.targets, 5)
+    ndcg10 += weight * binaryNdcgAt(ranked, sample.targets, 10)
+    ndcg20 += weight * binaryNdcgAt(ranked, sample.targets, 20)
+    ndcg50 += weight * binaryNdcgAt(ranked, sample.targets, 50)
     contextRecall += weight * contextRecallAtBudget(ranked, sample, 4096)
     meanReciprocalRank += weight * reciprocalRank(ranked, sample.targets)
     totalWeight += weight
@@ -92,6 +102,10 @@ export const evaluateCandidate = (snapshot, candidate) => {
       recallAt10: 0,
       recallAt20: 0,
       recallAt50: 0,
+      ndcgAt5: 0,
+      ndcgAt10: 0,
+      ndcgAt20: 0,
+      ndcgAt50: 0,
       contextRecallAt4096: 0,
       meanReciprocalRank: 0,
     }
@@ -101,6 +115,10 @@ export const evaluateCandidate = (snapshot, candidate) => {
     recallAt10: recall10 / totalWeight,
     recallAt20: recall20 / totalWeight,
     recallAt50: recall50 / totalWeight,
+    ndcgAt5: ndcg5 / totalWeight,
+    ndcgAt10: ndcg10 / totalWeight,
+    ndcgAt20: ndcg20 / totalWeight,
+    ndcgAt50: ndcg50 / totalWeight,
     contextRecallAt4096: contextRecall / totalWeight,
     meanReciprocalRank: meanReciprocalRank / totalWeight,
   }

benchmarks/retrieval/evaluation/promotion-evidence.ts

@@ -14,6 +14,10 @@ import type {
 const BOOTSTRAP_SAMPLES = 1_000
 const STABILITY_EPSILON = 0.005
 const QUALITY_METRICS: readonly QualityMetric[] = [
+  "ndcgAt5",
+  "ndcgAt10",
+  "ndcgAt20",
+  "ndcgAt50",
   "recallAt5",
   "recallAt10",
   "recallAt20",
@@ -124,7 +128,8 @@ const buildUncertainty = (rows: readonly PromotionHoldoutRow[]): readonly Holdou
 }
 
 const objectiveMetric = (objective: RouterObjective): QualityMetric => {
-  if (objective === "direct") return "recallAt5"
+  if (objective === "direct") return "ndcgAt5"
+  if (objective === "direct-recall-first") return "recallAt5"
   return objective === "reranker-top20" ? "recallAt20" : "recallAt50"
 }