compdemocracy · jucor · Jul 27, 2026
diff --git a/delphi/docs/CLOJURE_QUIRKS.md b/delphi/docs/CLOJURE_QUIRKS.md
@@ -29,7 +29,7 @@ Status legend: **REPLICATED** (Python legacy mode reproduces it) ·
 | Q8 | **Meta-tid `0`-is-truthy routing bug (#1961)** — every comment took the meta branch of `priority-metric`, flattening priorities to ~all-49. | math/.../repness or priorities path; see MATH_ALGORITHM_HISTORY.md | FIXED-UPSTREAM (#2611, merged 2026-07-18; not yet in a prod deploy as of 2026-07-21) | Done; watch prod deploy |
 | Q9 | **Subgroup smoother was UNCLAMPED in prod** until the #2575 clamp landed via #2609 (merged 2026-07-18). Only relevant to blobs generated by pre-#2609 workers. | conversation.clj:534-567 pre-#2609 | FIXED-UPSTREAM (and moot for us given Q7 carve-out) | — |
 | Q10 | **Large-conv mini-batch PCA is UNSEEDED-RANDOM** — `conv-update` dispatches to `large-conv-update` when n-ptpts > 10000 OR n-cmts > 5000 (conversation.clj:784-815); its `:pca` runs `partial-pca` on a fresh Mersenne-Twister row sample PER ITERATION with NO seed (`(sampling/sample (range n-ptpts) :generator :twister)`, conversation.clj:763-773) — so two Clojure runs of the same large conversation produce different PCA (and everything downstream). There is NO deterministic reference on this path, not even Clojure itself. Verified empirically 2026-07-22: pakistan/engage/bg2050 battery entries diverge at EXACTLY the first step crossing the cutoff (n-cmts 4914→6028, 4850→5765, 4694→6701) and match on every step before it. | conversation.clj:757-773 (sample-size + large graph), 784-815 (dispatch) | CARVED-OUT of R1 certification: the battery certifies the deterministic full-PCA path by pinning `{:ptpt-cutoff :cmt-cutoff}` HUGE in the replay drivers (conv-update accepts opts; math/dev/ change only — exclusion logged per run, never silent). Python keeps full PCA at all sizes. | Poller phase must decide the production-cutover story for large convs (python full-PCA is deterministic and strictly better; document as an intentional improvement over Clojure's randomized mini-batch). |
-| Q11 | **kmeans distances suffer dot-formula cancellation — near-coincident points TIE at exactly 0.0 and merge.** vectorz's `matrix/distance` on the row types kmeans actually passes (ArraySubVector view vs Vector center) computes d^2 = |a|^2+|b|^2-2ab, whose cancellation floors any true distance below ~1e-8 to EXACTLY 0.0 (verified in-process: pid-5/7 pair at true distance 4.66e-15 -> both cluster distances 0.0, while the same call with a copied row returns the true values). Ties then resolve via min-key LAST-wins -> near-coincident points collapse into the LATER cluster and the emptied cluster is dropped (vw every-vote step 57: cluster 8 absorbs [5 7], cluster 6 dropped). Python's norm(a-b) has no cancellation -> no tie -> no merge. | clusters.clj:44-52 (add-to-closest via matrix/distance); verified 2026-07-22 via math/dev/proj_probe.clj | TO REPLICATE (legacy mode): compute kmeans distances as sqrt(max(0, |a|^2+|b|^2-2ab)) in float64 — reproduces the 0.0 tie deterministically (numpy check confirms) | CORRECTED 2026-07-22 s4: `most-distal` does NOT use the cancellation formula — its rows come from `get-row-by-name` (matrix/get-row), a type on which `matrix/distance` returns the TRUE value (pc-revote-01 probe: clj's near-tie gaps are 2.5e-16/5.6e-17 = true-formula-sized, not the 1e-14 the cancellation formula produces on the same points). Python's port applies the cancellation formula at the shared helper, so py most_distal differs from clj in the low bits — benign wherever gaps exceed ~1e-8·scale; the only regime where it matters is the Q13 knife-edge class, which is carved out. Left as-is deliberately. NaN addendum (s7, #2663 review): the port lets NaN PROPAGATE through _euclidean (real vectorz has no clamp); only negative cancellation residue is floored to 0.0 — pinned by test_euclidean_propagates_nan_instead_of_clamping. |
+| Q11 | **kmeans distances suffer dot-formula cancellation — near-coincident points TIE at exactly 0.0 and merge.** vectorz's `matrix/distance` on the row types kmeans actually passes (ArraySubVector view vs Vector center) computes d^2 = |a|^2+|b|^2-2ab, whose cancellation floors any true distance below ~1e-8 to EXACTLY 0.0 (verified in-process: pid-5/7 pair at true distance 4.66e-15 -> both cluster distances 0.0, while the same call with a copied row returns the true values). Ties then resolve via min-key LAST-wins -> near-coincident points collapse into the LATER cluster and the emptied cluster is dropped (vw every-vote step 57: cluster 8 absorbs [5 7], cluster 6 dropped). Python's norm(a-b) has no cancellation -> no tie -> no merge. | clusters.clj:44-52 (add-to-closest via matrix/distance); verified 2026-07-22 via math/dev/proj_probe.clj | TO REPLICATE (legacy mode): compute kmeans distances as sqrt(max(0, |a|^2+|b|^2-2ab)) in float64 — reproduces the 0.0 tie deterministically (numpy check confirms) | CORRECTED 2026-07-22 s4: `most-distal` does NOT use the cancellation formula — its rows come from `get-row-by-name` (matrix/get-row), a type on which `matrix/distance` returns the TRUE value (pc-revote-01 probe: clj's near-tie gaps are 2.5e-16/5.6e-17 = true-formula-sized, not the 1e-14 the cancellation formula produces on the same points). Python's port applies the cancellation formula at the shared helper, so py most_distal differs from clj in the low bits — benign wherever gaps exceed ~1e-8·scale; the only regime where it matters is the Q13 knife-edge class, which is carved out. Left as-is deliberately. NaN addendum (s7, #2663 review): the port lets NaN PROPAGATE through _euclidean (real vectorz has no clamp); only negative cancellation residue is floored to 0.0 — pinned by test_euclidean_propagates_nan_instead_of_clamping. Vectorization addendum (s7, PR #2679): the batched-matmul columns preserve the cancellation ties BIT-EXACTLY (dgemv/einsum were rejected for last-ulp reassociation drift; exact-== pins travel with CI). |
 | Q12 | **Cold-tick PCA start vector is UNSEEDED-RANDOM** (`rand-starting-vec`, pca.clj:79-82 — the original author's own 'should really throw a [seeded] random number generator in the equation here... XXX' comment). With a small eigengap the fixed 100 power iterations do NOT fully converge, so the start-dependent residual (~1e-4 on pc-smallmix-01 step 0) survives into comps/projections — even two Clojure runs differ on the cold tick. Warm ticks are unaffected (start-vectors = previous comps). | pca.clj:79-101 | CARVED OUT for certification: BOTH replay drivers pin the cold start to the ONES vector — the exact value power-iteration already pads new-comment columns with (pca.clj:46-49 / pca.py _power_iteration) — via a single-element [1.0] start that padding expands to any width (dev/replay.clj certify-cold-start-pca + replay/driver.py seed). Logged per run. | Production: seed the start vector (the author's own XXX) — deterministic cold ticks with no behavior change at convergence |
 | Q13 | **Warm-chain split-loop extraction order is knife-edge-chaotic on tie-dense geometry.** `clean-start-clusters`' split loop (clusters.clj:250-273) extracts the most-distal point one at a time; when several near-coincident candidates tie (within-engine distance gaps at the few-ulp level, e.g. 2.5e-16/5.6e-17 on pc-revote-01 step 1 among 4 pids), the extraction ORDER — and thus the minted singleton ids and final partition — is decided by sub-ulp arithmetic noise. Cross-engine, projections differ at ~1e-5 (residual small-eigengap power-iteration noise, tolerant-accepted), ELEVEN orders above the gaps: no arithmetic replication can reproduce Clojure's order (verified 2026-07-22: both probes extract the identical 28-pid sequence, then clj picks {82,99} where py picks {80,99} from the tied set {80,82,99,108}; even the true-distance formula ranks them differently per engine). Same irreducibility class as the vw every-vote step-57 knife edge. | clusters.clj:202-217 (most-distal), 250-273 (split loop); probes: math/dev/proj_probe.clj split-probe + delphi/scratch/probe_revote_split.py | CARVED OUT: battery keeps revote coverage via pc-revote-02 (34 ptpts, ~28% revotes — rich geometry, no knife edge; MATCH 6/6 first try) in place of pc-revote-01 (205 ptpts on 15 comments — near-discrete projection space). Ledgered on FP-912391ece7/FP-c29173e1ba/FP-98dc728043. | Not a Clojure bug — an irreducible float-chaos regime. Any future dataset that diverges ONLY in split-loop extraction order on few-ulp gaps belongs to this class: probe with split-probe, then swap or document. |
 | Q14 | **No small-dimension guards — tiny matrices run the real math.** Clojure runs powerit PCA on 1x1/1xN/Nx1 rating matrices (real center, comps rank-capped at min(rows,cols)) and `conv-repness` always returns its best-agree comment even for degenerate shapes — where pre-parity Python short-circuited: PCA returned zeros (center -0.0 vs clj -1.0 on a 1x1 matrix; comps zero-padded to 2 vs rank-capped 1) and `conv_repness` returned empty repness/consensus below shape 2. | pca.clj (powerit-pca, no dim guard); repness.clj conv-repness (best-agree fallback) | REPLICATED (2026-07-22, PR #2653: legacy mode relaxes the `<2` guards in conversation.py::_compute_pca, pca.py::pca_project_dataframe, repness.py::conv_repness; single-vote fixtures with exact Clojure-derived values, tests/test_legacy_blob_shape.py; certified end-to-end by the vw-every-vote-56 battery entry whose early steps are 1xN). Ledgered late — flagged by the 2026-07-22 s4 review pass; row added then. | `improved` mode keeps the guards (sane early returns) |

diff --git a/delphi/docs/CUTOVER_RUNBOOK.md b/delphi/docs/CUTOVER_RUNBOOK.md
@@ -1,19 +1,26 @@
 # Clojure→Python math cutover runbook
 
-Written 2026-07-26 (post R1-parity DONE, GOAL_STATE.md). Companion:
-MATH_POLLER_DESIGN.md §4 (phases), MATH_POLLER_EQUIV_SPEC.md (the live
-equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19).
-
-## Evidence base (what is PROVEN as of 2026-07-26)
-
-- R1 battery: 20/20 MATCH on four consecutive clean pass-pairs (2026-07-24),
-  ledger zero open. Warm chains, moderation, meta, bans, revotes, degenerate
-  ticks, restart seams.
-- Live poller equivalence vs the REAL Clojure container: vw 8/8 batches +
-  pc-meta-02 6/6 batches MATCH (moderation stream live, kill+restart seam,
-  tick/watermark semantics, bidToPid/ptptstats row-identical; floats within
-  the measured clj self-jitter envelope).
-- Full delphi suite green (1134 passed at goal close; CI green at stack tip).
+Written 2026-07-26 (post R1-parity DONE); refreshed 2026-07-28 (s7, post
+GOAL_CUTOVER_READY DONE). Companion: MATH_POLLER_DESIGN.md §4 (phases),
+MATH_POLLER_EQUIV_SPEC.md (the live equivalence protocol),
+CLOJURE_QUIRKS.md (Q1-Q19), POST_CUTOVER_IMPROVEMENTS.md (the queue).
+
+## Evidence base (what is PROVEN as of 2026-07-28, s7)
+
+- MODE COLLAPSE LANDED (#2665-#2671): the engine has ONE code path —
+  exact legacy semantics; flag machinery deleted; improved branches
+  parked (improvements/* bookmarks); bans deleted (not a Polis feature).
+- Battery: 20/20 MATCH pairs re-certified at every s7 milestone —
+  post-collapse, post-refactor (#2673), post-vectorization (#2679), and
+  on the final tree; divergences ledger 81 entries, 0 open.
+- Live poller equivalence vs the REAL Clojure container, RE-RUN on the
+  collapsed tree: vw 8/8 + pc-meta-02 8/8 batches MATCH, non-vacuous,
+  0 envelope-excused divergences (moderation stream, kill+restart seam,
+  bidToPid/ptptstats row-identical).
+- Goldens re-recorded at the collapse tree (verify-then-record; comparer
+  7/7 PASS); full delphi suite green (1171 passed at s7 close).
+- Clarity refactor 14b/14c (#2673) + vectorized warm-start kmeans
+  (#2679, bit-identical, warm tick ~70x) landed pre-cutover.
 
 ## What is NOT yet proven (risk register)
 
@@ -26,16 +33,21 @@ equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19).
    (documented improvement, same blob shape, server-compatible). A shadow
    comparer WILL flag these convs — expected, not a defect. Decide the
    acceptance for them up front (structural-only, or exclude from compare).
-3. **Poller throughput**: ~1.66 ticks/s per process — measured on EC2
-   (r8g.4xlarge, cost-model study) on biodiversity-sized replays.
-   **PRE-FLIP MEASUREMENT DONE (2026-07-27, s7)** — one full-PCA tick of
-   the largest prodclone shape (33,422 ptpts x 783 cmts, 2.0M votes;
-   synthesized, seeded) on r8g.4xlarge via
-   scripts/large_conv_tick_bench.py:
-   **cold tick 519.6s (~8.7 min); WARM (steady-state) tick 1856.0s
-   (~30.9 min)** — the warm tick is ~3.6x the cold one (legacy kmeans
-   lineage warm-start dominates). Local M-series cross-check: 430.8s /
-   2095.3s — same order, so this is algorithmic, not instance-bound.
+3. **Poller throughput**: ~1.66 ticks/s per process on biodiversity-sized
+   replays — measured on EC2 (r8g.4xlarge, cost-model study) with the
+   PRE-VECTORIZATION engine, so it is now a stale LOWER BOUND (#2679
+   speeds up every conv's k-means, not just giants; re-measure during
+   the shadow soak if a capacity number is needed).
+   **PRE-FLIP MEASUREMENT (2026-07-27 s7, pre-vectorization —
+   SUPERSEDED by the verdict below)** — one full-PCA tick of the largest
+   prodclone shape (33,422 ptpts x 783 cmts, 2.0M votes; synthesized,
+   seeded) on r8g.4xlarge via scripts/large_conv_tick_bench.py:
+   cold tick 519.6s (~8.7 min); WARM (steady-state) tick 1856.0s
+   (~30.9 min). The then-observed "warm ~3.6x cold" asymmetry was an
+   artifact of the un-vectorized port's per-pair python loops in the
+   lineage warm start — ELIMINATED by #2679 (post-vectorization the two
+   are within ~10%: 29.0s vs 26.6s). Local M-series cross-check ran
+   same-order both times (430.8s/2095.3s before; 28.2s/26.7s after).
    **VERDICT (FINAL, 2026-07-28 s7): serial is OK at every observed
    shape.** Item 9a (vectorized warm-start k-means, PR #2679 —
    bit-identical: exact-== pins vs the scalar reference, knife-edge Q11
@@ -66,14 +78,57 @@ equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19).
    python's per-zid FIFO+lock design does not have it (equivalence runs
    verified py carries the full vote stream).
 
+## Execution shape (s7 rulings + analysis — read before Step 0)
+
+**Shadow vs clean replace (analysis 2026-07-28; decision pending
+Julien):** recommended = TIME-BOXED SHADOW, 24-48h, exit checklist
+below. Rationale: it tests the only untested dimension (real prod
+churn/concurrency/dirty data) at near-zero complexity — the service,
+env var, and compare machinery all exist; the time box kills
+shadow-limbo risk. Clean replace is defensible on the evidence
+(bit-exact battery + live equivalence) and rollback stays cheap
+(restart `math`; caching_tick is MAX+1 both ways), but forfeits the
+baseline rows that make subtle math weirdness detectable. Memory is a
+non-issue either way: host 128 GiB, python capped 16g (set
+MATH_CONV_CACHE_CAP — and keep it set in ANY long-running deployment,
+not just the soak; eviction cost = the certified restart seam).
+Shadow exit checklist (agree BEFORE starting): rows advancing on all
+active zids; zero parked zids / errorconv dumps; spot-compare N active
+zids structurally identical (poller_equiv comparer on row pairs);
+large-conv divergence dismissed per risk 2/Q10.
+
+**One WIP PR per step (for a future session):**
+- PR-S0 (promote): get the stack onto `stable` (prod deploys track
+  stable, not edge — after_install.sh pulls stable).
+- PR-S1 (shadow): scripts/after_install.sh math role line →
+  `up -d math math-python`; add MATH_CONV_CACHE_CAP + MATH_PYTHON_ENV
+  to the SSM-sourced .env (polis-web-app-env-vars secret); exit
+  checklist copied into the PR body.
+- PR-S2 (flip): ONE mechanism (ruling needed: poller MATH_ENV→'prod' vs
+  server mathEnv→'python'); revert instructions in the PR body.
+- PR-S3 (decommission): remove `math` from compose + its
+  after_install.sh line; archive note for the Clojure tree.
+
+**CDK impact: NONE required for steps 0-3.** The python poller runs on
+the existing math-worker host (r8g.4xlarge, MathWorkerLaunchTemplate)
+via compose; same Postgres path/security groups; math_writer needs no
+new IAM (Postgres only); CodeDeploy math deployment group unchanged
+(the only deploy-side edit is after_install.sh, which ships with the
+repo). Verified: cdk/ec2.ts, cdk/launchTemplates.ts, appspec.yml,
+scripts/after_install.sh. CDK would only enter later if the math host
+itself is retired/resized post-decommission (candidate: downsize
+r8g.4xlarge once the vectorized engine's real utilization is known —
+measure during the soak first).
+
 ## Step 0 — land the stack (morning)
 
-1. Triage the 18 Copilot reviews requested overnight (2026-07-26) on
-   #2641-#2658; apply/reply per the standing triage rules.
-2. Confirm CI green: stack-tip python-ci dispatch + PR checks (see
-   spr status; #2648's mid-stack red is a stack-position artifact — the
-   same tests pass from #2656 upward — cosmetic for deploy, which builds
-   the tip).
+1. Reviews: DONE through #2682 (Copilot triage s6 = #2663; Copilot
+   credits exhausted since — all later PRs reviewed by independent
+   review agents, all sound; findings applied). Nothing outstanding.
+2. Confirm CI green at the stack tip (python-ci workflow_dispatch on the
+   tip branch; every s7 dispatch was green). Historic note: a mid-stack
+   red (e.g. #2648-era) is a stack-position artifact — deploy builds the
+   tip.
 3. Merge bottom-up: `jj spr merge --count <N>` (spr handles squash order).
    NEVER the GitHub UI. Then a normal edge deploy.
 
@@ -115,7 +170,7 @@ small/mid convs; large-conv divergence understood per risk #2.
 One env change, instantly reversible:
 - Set the python poller's MATH_ENV to the server's Config.mathEnv ('prod');
   stop the clojure `math` service. (Or flip the server's MATH_ENV to
-  'delphi' — pick ONE mechanism and write it down.)
+  'python' — pick ONE mechanism and write it down.)
 - Watch: TS prefetch (pca.ts caching_tick > last, ~2.5s poll) keeps
   serving; nextComment routing gets comment-priorities; participants
   bidToPid present.
@@ -125,7 +180,9 @@ coexist; nothing is destroyed by the flip in either direction.
 
 ## Step 3 — decommission (later)
 
-Remove the `math` service from compose/deploy; archive the Clojure tree
-(it remains the R1 oracle). Follow-ups parked in the journal: equiv-in-CI
-decision, improved-mode ban coverage, fraction-cut py-round fix, quirk
-un-replication in improved mode (the post-cutover engine option).
+Remove the `math` service from compose/deploy (and its `up -d math`
+line in scripts/after_install.sh); archive the Clojure tree (it remains
+the certification oracle). Follow-ups now live in
+POST_CUTOVER_IMPROVEMENTS.md (items 2-9b, 11, 12 — quirk un-replication,
+warm-start persistence, optional seeded sampled PCA) plus the journal's
+equiv-in-CI decision and the fraction-cut py-round fix.
diff --git a/delphi/docs/MATH_POLLER_DESIGN.md b/delphi/docs/MATH_POLLER_DESIGN.md
@@ -82,7 +82,7 @@ math_env string written), `VOTE_POLLING_INTERVAL` (ms, default 1000),
 1. **Shadow (this PR):** new compose service `math-python` (profile-gated,
    `--profile math-python`; renamed s7 from delphi-math-poller) running
    next to the Clojure `math` service, writing under a
-   DIFFERENT `math_env` (e.g. `MATH_ENV=delphi` while Clojure writes `prod`/`dev`).
+   DIFFERENT `math_env` (e.g. `MATH_ENV=python` while Clojure writes `prod`/`dev`).
    `UNIQUE(zid, math_env)` makes the rows invisible to the prod server. No consumer
    change, zero production risk.
 2. **Parity monitoring:** a comparer job diffs Python-vs-Clojure `math_main` rows per