From f0bb734c6551947029b94ae7f8300173f2687104 Mon Sep 17 00:00:00 2001 From: Julien Cornebise Date: Mon, 27 Jul 2026 18:45:00 +0200 Subject: [PATCH] =?UTF-8?q?python-math=20#43:=20docs(delphi):=20s7=20goal?= =?UTF-8?q?=20state=20+=20journal=20=E2=80=94=20mode=20collapse=20executed?= =?UTF-8?q?,=20battery=2020/20?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Session 7 wind-down for `GOAL_CUTOVER_READY.md` (the standing goal document for the Python math cutover). `GOAL_STATE.md` is rewritten: Phases 0-2 done; next up are reviews triage, the clarity refactor, goldens + gates, and the EC2 measurement. The journal carries the Phase 0 A/B data, the Phase 1 inventory, the Phase 2 execution record, and the post-collapse evidence from the certification battery (which replays 20 recorded dataset entries through the Python engine and compares against Clojure reference recordings): 20/20 MATCH on a full re-replay plus a cached pass. commit-id:29a11a32 --- delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md | 49 +++++++++++++++ delphi/docs/GOAL_STATE.md | 83 +++++++++++++------------ 2 files changed, 92 insertions(+), 40 deletions(-) diff --git a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md index 801ad0a3a..7167730ee 100644 --- a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md +++ b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md @@ -4213,3 +4213,52 @@ Also purged post-suite: scripts/poller_equiv.py CLI --engine-mode plumbing docker-compose delphi-math-poller env line, example.env comment, CLOJURE_QUIRKS preamble + MATH_POLLER_DESIGN updated to collapse-era wording (historical spec docs left as records). + +### s7 wind-down — What's Next + +Pushed: PRs #2665-#2671 (collapse series) + #2664 (Phase 0 battery +speedup). python-ci dispatched on spr/edge/7f42df81 (run 30286254481); +collapse-series review subagent launched; Copilot requested once on +#2659/#2663 — ALL to be checked at next orientation. Parks live on jj +bookmarks improvements/item-{2,4,5,8} (pushed to origin). Remaining +phases: 3 (clarity refactor 14c/14b), 4 (goldens + double battery pass + +equiv gate), 5 (EC2 measurement). See GOAL_STATE.md for numbered actions. + +## Session 7 (cont.): Phase 3 clarity refactor SHIPPED — battery bit-identity ×2 trees + +- **14b**: TestBlobInjectionStats (tests/test_repness_unit.py) — Clojure + blob group memberships (unfolded via the blob's own base-clusters) + the + dataset votes injected into the PRODUCTION stats path; every blob repness + entry compared per (gid, tid) on n-success/n-trials/p-success/p-test/ + repness/repness-test/repful-for. GREEN on vw + biodiversity. Gotcha: + test-conversation matrices carry STRING pids/tids vs the blob's ints + (map on the way in); Clojure emits repness-test ROUNDED (~7 sig digits) + → that one field compares at rtol 2e-6, the rest at 1e-9. +- **14c**: compute_group_comment_stats_df → + _group_comment_vote_counts (plumbing) + _comment_stats_from_counts + (recipe). Pure code motion. **Battery on the refactored tree: 20/20 + MATCH (full py re-replay, 18m46s)** — bit-identity PROVEN. Full suite + 1155 passed (+2 blob pins, -1 deduped parametrize, one env-gated skip). +- Shipped as PR #2673 (spr/edge/acff8fbe); python-ci dispatched (run + 30288678922); review subagent launched. jj gotcha hit: `jj split` + opens an editor (use JJ_EDITOR=true) and gives BOTH halves the original + description INCLUDING the spr commit-id trailer, and the spr bookmark + follows the working copy — rewrite the second half's description fresh + and `jj bookmark set -r --allow-backwards`. +- Collapse-series review agent verdict: CLEAN (no high-confidence + findings; verified every branch reduction = the legacy arm, fingerprint + freezing exact, no test-expectation drift). Its 3 sub-threshold + cleanups applied (runbook env line, deduped parametrize, docstrings). + Phase 0 PR #2664 python-ci: SUCCESS. + +### Phase 4 diagnostic (goldens) + +`scripts/regression_comparer.py` on the collapsed tree: NO golden +snapshots exist for the public datasets (vw/biodiversity — never +recorded in this worktree); the 5 private-dataset goldens live under +real_data/.local/*/golden_snapshot.json. That's why the suite stayed +green through the collapse — golden comparisons skip without snapshots. +Phase 4 re-record therefore = private goldens (--include-local) + +optionally recording public ones; VERIFY against the battery's certified +clj recordings BEFORE recording (never blind). Recorder: +scripts/regression_recorder.py. diff --git a/delphi/docs/GOAL_STATE.md b/delphi/docs/GOAL_STATE.md index ec04dafde..d9dfe1976 100644 --- a/delphi/docs/GOAL_STATE.md +++ b/delphi/docs/GOAL_STATE.md @@ -2,50 +2,53 @@ STATUS: IN PROGRESS # GOAL_STATE — checkpoint for GOAL_CUTOVER_READY.md (cap ~50 lines) -Predecessor GOAL_R1_PARITY.md: ACHIEVED 2026-07-24 (evidence pointers in its -final journal entries; battery ×4 clean pairs, live equiv PASS ×2 datasets). - -## Where we are (2026-07-27, session 6 wind-down) - -- Copilot triage COMPLETE: 25 threads — 16 fixes applied (TDD; 647-test gate - green), 2 quirk-rejections cited, 3 declines, 1 deferral (#2644). Battery - ×2 re-ran on the triaged tree (s6 logs — check verdicts at orientation if - this session ended before they landed). Issues #2660/#2661/#2662 opened - (Q10/Q12/Q13+Q18 — "fixed by the Python push"). -- Julien rulings captured in POST_CUTOVER_IMPROVEMENTS.md + - CUTOVER_RUNBOOK.md: bans DROPPED entirely; equiv = release gate not CI; - sharding parked (data: p99=5 active convs/min vs ~100 ticks/min EC2- - measured serial capacity); clarity refactor moved PRE-cutover; large-conv - EC2 tick measurement is a flip precondition; run_delphi.py is - PRODUCTION-called (POST /api/v3/delphi/jobs → job_poller FULL_PIPELINE). -- Battery timing DATA (journal s6): first pass after an engine change ≈36 - min (py re-replay, clj cached); second consecutive pass ≈21 SECONDS; - full clj re-record ≈91 min. Phase 0 tooling attacks the 36-min pass. +## Where we are (2026-07-27, session 7 wind-down) + +- Phase 0 DONE + A/B-proven: engine-scoped py cache hash + 6-way parallel + battery (PR #2664, python-ci green, review clean). Costs now: no edit + ~22s, harness-only edit ~2m, engine edit ~19m (long pole + pakistan:uniform8 ~18m — <8m target unreachable without intra-entry + parallelism; journal s7). +- Phase 1 DONE: engine_mode inventory journaled (s7). +- Phase 2 DONE — MODE COLLAPSE EXECUTED: 7 commits = PRs #2665-#2671. + DONE-gate grep = 0 hits over delphi/polismath/. Ban filtering DELETED + outright. Parks minted: jj bookmarks improvements/item-{2,4,5,8} + (verbatim reverse patches off the C7 commit; NOT buildable — they + carry old flag refs; re-landing = keep the improved side only). + Battery on the collapsed tree: 20/20 MATCH full py re-replay (19m21s) + + cached pass 22s. Full suite 1155 passed / 22 skipped / 44 xfailed + / 2 XPASS (D9/D10 vw-cold_start now match Clojure — parity improved). +- Phase 3 DONE: 14b blob-injection pins + 14c two-phase split (PR + #2673); battery 20/20 on the refactored tree (bit-identity); collapse + review agent verdict CLEAN, its 3 cleanups applied. #2664 CI green. ## Next actions -1. s6 battery CONFIRMED clean: 20/20 MATCH ×2 on the triaged tree - (battery_s6_pass{1,2}.log; only 3 clj re-records fired — the restart - entries carry no comments CSV). Commit split DONE: triage fixes = PR - #2663 (spr/edge/efb914d7), docs = #2659. python-ci dispatched on - #2663's head (run 30276844539) — CHECK at orientation. Thread - replies/resolves: CONFIRMED complete — all 20 open threads across 11 - PRs replied (citing #2663) and resolved; 25/25 Copilot comments closed. -2. Phase 0 of GOAL_CUTOVER_READY.md: battery tooling speedup (hash scoping - + parallel entries; A/B-prove — timing data in journal s6: 36-min - first pass / 21-s cached pass). THEN Phase 1 inventory (grep all - engine_mode branch sites; classify DELETE/PARK/KEEP; journal it). -3. Review the new PRs per protocol when review-ready: #2659 (docs), #2663 - (triage batch — request Copilot once; our agent already covered the - content via the triage itself). -4. Parked questions needing input: none — all rulings recorded. +1. Orientation: check CI runs 30286254481 (collapse tip) + 30288678922 + (#2673 14b/c) + the #2673 review agent result; Copilot comments on + #2659/#2663. Triage per protocol (a "fix" undoing legacy semantics = + post-cutover queue item). Phase 3 is DONE (PR #2673; battery 20/20 on + the refactored tree = bit-identity; journal s7 cont.). +2. Phase 4: goldens. Diagnostic done (journal s7 cont.): public datasets + have NO goldens (comparisons skip); the 5 private goldens are at + real_data/.local/*/golden_snapshot.json. Verify current private-golden + drift with `uv run python scripts/regression_comparer.py --include-local` + (needs /private-data setup), cross-check intended new values against + the battery clj recordings, THEN re-record via + scripts/regression_recorder.py; full suite --include-local green. +3. Phase 4 gates on the final tree: TWO consecutive clean battery passes + + poller_equiv.py full-run PASS (non-vacuous) on vw AND pc-meta-02 + (pgproxy 127.0.0.1:15432 up: `docker start polis-dev-postgres-1 + pgproxy`; docker compose services for the clj container). +4. Phase 5: EC2 large-conv tick measurement (bench AWS profile, 33k + ptpts × 783 cmts shape) → number + serial-OK verdict into + CUTOVER_RUNBOOK.md risk register. +5. Only after all of 1-6 of the DONE list hold on the final tree: flip + line 1 here to STATUS: DONE. ## Pointers - Contract: GOAL_CUTOVER_READY.md. Roadmap: POST_CUTOVER_IMPROVEMENTS.md. - Runbook: CUTOVER_RUNBOOK.md. Quirks: CLOJURE_QUIRKS.md Q1-Q19. -- Battery: scripts/certify.py run (20 entries). Equiv release gate: - scripts/poller_equiv.py full-run (needs pgproxy 127.0.0.1:15432 up: - `docker start polis-dev-postgres-1 pgproxy`). -- Suite baseline: 1134 passed / 22 skipped / 46 xfailed (2026-07-24) + s6 - additions; gate delegation protocol in GOAL_R1_PARITY.md. +- Battery: cd delphi && uv run python scripts/certify.py run (workers=6). +- Suite baseline: 1155/22/44 + 2 xpassed (s7 collapse tree). +- s7 journal: Phase 0/1/2 + battery evidence (CLJ-PARITY-FIXES-JOURNAL.md).