fix(heldout): declare score-reliability sample size

[seonghobae]

Stack

Draft successor of #1099 (b975c3dd). Parents #1098, #1097, #1096, #1095, #1093, #1092, #1091, #1090, #1074, and psychometric KPI owner #1067 stay open. This PR does not close #1099.

Exact head: 9ff12b5dfbf2a36f21f0e16cc311eb3949b90e94.

Problem

Held-out posterior reliability evidence hid RELIABILITY_SAMPLE_SIZE = 1_200, so weak- versus strong-information cases used a repository-selected person population.

Change

• Require declared sample_size on _validate_score_reliability. Missing, boolean, or non-positive values fail closed.
• The harness run still writes 1,200 as this run's choice and records sample_size_per_case.
• ADR 0051 is Proposed. Production route/conduct defaults stay locked.

Verification

Local smallest tests:

• 64 passed in 16.19s (tests/test_psychometric_routing.py, tests/test_psychometric_benchmark_boundaries.py, tests/test_planning_adr_identifiers.py)
• interrogate on the held-out script: 100%

This is declaration-contract evidence, not buyer-held-out accuracy, p95 latency, or protected merge.

Remaining no-heuristics work

Other harness sample sizes (judge, exploration rate) remain later slices. No production policy or release is authorized. Parent #1067 still needs independent review.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Normal predecessor integration at 3e85d32 retains original reliability sample declaration 9ff12b5 and includes predecessor da4ceac. No-detection censoring, alarm/detection denominators, null ineligible calibration candidate and retained calibration results now propagate through this stack. Merged-tree validation: 31 boundary/ADR identifier tests passed in 12.92 seconds; Ruff and diff whitespace checks passed; CodeGraph indexing completed. No force push or discarded delta. This is local regression evidence, not real-data accuracy, protected merge or release evidence. Required checks and reviews must use the new head.

[seonghobae]

Superseded by origin/main after #1101 merged (worker restack check).

Evidence:

• PR head 3e85d321 is an ancestor of origin/main @ 62d74579 (Merge fix(heldout): declare judge-effect sample size #1101).
• Commits c7964228, 8533a602, 9ff12b5d are in main; declared reliability sample-size API + ADR 0051 present.
• Focused tests on main: test_score_reliability_requires_declared_sample_size and test_score_reliability_uses_declared_sample_size — 2 passed in 1.57s.
• Did not force-push head==base; did not admin-merge.