diff --git a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md index 1feae166b..65b52abc9 100644 --- a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md +++ b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md @@ -4496,3 +4496,17 @@ bit-identical) + #2680 (math-python rename) pushed; CI dispatched (run completion messages need JSON parsing (tab-split receipt handles broke delete → stale redelivery); the vectorized run's job label says large-conv-tick (sed missed escaped quotes) — S3 key disambiguates. + +### s7 close: cutover handoff shipped + one spr incident (recovered) + +HANDOFF_CUTOVER_EXECUTION.md = entry point for the cutover-PRs session +(PR #2684): state summary, the two open rulings (shadow-vs-replace; +flip mechanism), per-PR specs S0-S3 incl. the Secrets-Manager env +reality, and the gotcha list. Runbook carries the canonical "Execution +shape" section (shadow analysis + exit checklist + no-CDK verdict). +Incident: re-describing the audit commit without preserving its +commit-id trailer forked a duplicate PR (#2682/#2683) — the EXACT +failure the 2026-07-17 memory warns about; recovered per its recipe +(kept the trailer-matching #2683, closed #2682, deleted the stray +bookmark+branch). A malformed `spr/edge/` bookmark from an +empty-commit spr update was also deleted. diff --git a/delphi/docs/HANDOFF_CUTOVER_EXECUTION.md b/delphi/docs/HANDOFF_CUTOVER_EXECUTION.md new file mode 100644 index 000000000..54422b3eb --- /dev/null +++ b/delphi/docs/HANDOFF_CUTOVER_EXECUTION.md @@ -0,0 +1,88 @@ +# HANDOFF: execute the Clojure→Python math cutover (steps 0-3 as WIP PRs) + +Written 2026-07-28 at the close of s7 (GOAL_CUTOVER_READY: DONE). This is +the ENTRY POINT for the session that ships the cutover. Read order: +1. This file. +2. CUTOVER_RUNBOOK.md — canonical: evidence base, risk register with the + FINAL measured verdict, "Execution shape" (shadow analysis, PR plan, + CDK verdict), steps 0-3. +3. GOAL_STATE.md (STATUS: DONE + walkthrough) if provenance is needed. + +## Where things stand (evidence all in-repo) + +- Engine: ONE code path, Clojure-exact legacy semantics. Certified: + battery 20/20 MATCH pairs at every s7 milestone incl. the final tree; + live equivalence vw 8/8 + pc-meta-02 8/8 non-vacuous; goldens + re-recorded (comparer 7/7); suite 1171 green. +- Performance solved: vectorized warm-start k-means (#2679, + bit-identical). r8g.4xlarge, 33,422×783/2.0M votes: cold 29.0s, warm + 26.6s (was 519.6s / 1856.0s). Verdict: serial OK at every shape; no + blocklisting (Julien ruling: never blocklist; warm start stays). +- Naming: compose service `math-python`, profile `math-python`, env + `MATH_PYTHON_ENV` (default math_env value 'python'). +- The ENTIRE stack is Draft/UNMERGED (spr-managed, ~#2613-#2682+). + Nothing is on `edge` yet, and prod deploys from `stable`. + +## OPEN RULINGS — get from Julien before the relevant PR + +1. Shadow vs clean replace. Recommendation on file (runbook "Execution + shape"): time-boxed shadow 24-48h with the written exit checklist. +2. Flip mechanism — pick ONE: poller MATH_ENV→'prod' vs server + mathEnv→'python'. Runbook step 2 demands it be written down. + +## The PRs + +- **PR-S0 — land + promote.** (a) Merge the stack bottom-up: + `jj spr merge --count ` — NEVER the GitHub UI (spr can't track UI + squashes). Julien decides the merge moment/team sign-off. (b) Promote + edge→stable: CHECK FIRST how stable has historically been advanced + (`git log origin/stable` — fast-forward vs PR; not verified in s7). + Prod's after_install.sh does `git reset --hard origin/stable`. +- **PR-S1 — shadow wiring.** scripts/after_install.sh, math role branch + (`elif [ "$SERVICE_FROM_FILE" == "math" ]`, ~line 105-108): change + `up -d math` → `up -d math math-python`. Env: MATH_PYTHON_ENV + + MATH_CONV_CACHE_CAP must reach the instance .env — that comes from + Secrets Manager `polis-web-app-env-vars` (AWS-side edit, needs + Julien/elevated creds — NOT bench, NOT a repo change; coordinate). + Copy the exit checklist (runbook Execution shape) into the PR body. + MATH_CONV_CACHE_CAP: set it (LRU; eviction cost = certified restart + seam). Pre-soak verify item: read the clj container's actual -Xmx on + the host (`docker stats`); memory math says 128 GiB host / 16g python + cap / clj unchanged — wide margins, but cite real numbers. +- **PR-S2 — flip.** One env change per ruling 2; revert instructions in + the PR body. Rollback semantics: both envs' rows coexist + (UNIQUE(zid, math_env)); caching_tick is MAX+1 so monotonicity + survives swaps in both directions; restart clj `math` to roll back. +- **PR-S3 — decommission.** Remove `math` from docker-compose.yml AND + its `up -d math` line in after_install.sh; archive note for math/ + (the Clojure tree stays as the certification oracle — do NOT delete). + +## Gotchas that will bite (all learned the hard way) + +- spr: one commit = one PR on the single stack bookmark. NEVER + `jj squash -m` into an spr commit (wipes the commit-id trailer → + garbage PRs); preserve trailers when re-describing; use + `--use-destination-message`. jj split gives BOTH halves the trailer — + rewrite the second half's description fresh and move the spr bookmark + back (`jj bookmark set spr/edge/ -r --allow-backwards`). +- jj-colocated: NEVER `git checkout --`/`git restore` a working file + (git index = parent commit; wipes uncommitted work). +- Copilot review credits are EXHAUSTED — use independent review-agent + subagents per PR (the s7 pattern; all 15+ s7 PRs reviewed that way). +- python-ci on spr branches needs manual `gh workflow run python-ci.yml + --ref ` (dispatch at wind-down, check at next orientation). +- Compose profiles gate DEV only; prod starts services BY NAME. +- Battery cost model: no engine edit ~22s cached pair; engine edit + ~19min re-replay (run `cd delphi && uv run python scripts/certify.py + run`). The cutover PRs touch deploy/compose only → cached pairs. +- Shadow comparer WILL flag the 7 historical large convs (Q10: clj rows + are unseeded-random there) — expected, not a defect (runbook risk 2). + +## Post-cutover (not this session, but adjacent) + +POST_CUTOVER_IMPROVEMENTS.md is the queue: parks on improvements/* +bookmarks (items 2/4/5/8), item 12 (persist warm-start state — kills +restart-induced K flips), item 9b optional. Candidate after the soak: +downsize the math host (CDK cdk/ec2.ts instanceTypeMathWorker) once +real utilization is measured — the vectorized engine likely doesn't +need an r8g.4xlarge.