python-math #47: docs+test(delphi): s7 conditions 4-6 evidence — EC2 measurement, equiv PASSes, NaN pin

delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md

@@ -4288,3 +4288,126 @@ PR #2673 review agent verdict: CLEAN — split proven behavior-preserving
 (including an equivalence proof of the counts_df.empty seam), blob test
 non-vacuous (45+42 entries), tolerances justified against Clojure's
 `(float repness-test)` cast at repness.clj:187.
+
+### Conditions 4+5 EVIDENCE (s7 cont.)
+
+- **Battery pair** (condition 4): 20/20 MATCH ×2 consecutive on the
+  Phase-4 tree; divergences.json = 81 entries, 0 open.
+- **Equivalence release gate** (condition 5): poller_equiv.py full-run
+  re-run LIVE on the collapsed+refactored tree —
+  vw: verdict PASS/MATCH, 8/8 batches, ticks OK both envs, 0
+  envelope-excused divergences (worst self-jitter 4.9e-06);
+  pc-meta-02: verdict PASS/MATCH, 8/8 batches, 0 envelope-excused
+  (worst self-jitter 1.5e-06). Evidence refreshed in place under
+  real_data/.local/replays/poller_equiv/{vw,pc-meta-02}/.
+- CI: run 30286254481 (collapse tip) SUCCESS; run 30288678922 (#2673)
+  SUCCESS; run 30302469207 (#2675 tip) dispatched.
+- PRs #2674 (FLI xfail + journal) + #2675 (large-conv bench tool)
+  pushed. Public golden_snapshot.json now gitignored (goldens stay
+  local artifacts; a 350k-line accidental snapshot was stripped from
+  #2674 before push).
+
+### Phase 5 IN FLIGHT (s7 cont.)
+
+EC2 measurement launched: i-057c881e212b2671d (r8g.4xlarge, us-east-1,
+bench profile, Project=polis-cost-model), self-terminating user-data
+(clone spr/edge/5747f8c9 → minimal venv (numpy/pandas/sklearn/natsort/
+click/pyyaml) → scripts/large_conv_tick_bench.py full 33k×783 shape →
+S3 results/large-conv-tick/ → SQS polis-cost-model-done → shutdown;
+dead-man 180 min; terminate-on-shutdown). Local full-size run in
+parallel on the M-series laptop for a comparison point. Smoke numbers
+(2000×200×120k, laptop): cold 6.5s, warm 14.4s — warm is the expensive
+side (legacy kmeans warm-start path dominates).
+
+### Review protocol correction (Julien, s7): Copilot credits exhausted
+
+The Copilot reviews requested on #2659/#2663 never ran — monthly AI
+credits are exhausted again. Per Julien: use INDEPENDENT /code-review
+subagents instead. Two launched (one per PR); do not re-request Copilot
+this month. (The collapse series #2665-#2671 and #2673 already had
+independent review-agent passes — both CLEAN.)
+
+### Phase 5 — LOCAL full-size result + #2659 review disposition (s7 cont.)
+
+Local (M-series laptop, arm64), 33,422 × 783, 2,005,320 votes:
+ingest 2.4s, **cold tick 430.8s (~7.2 min), WARM tick 2095.3s (~35 min)**.
+The warm tick — the steady per-tick cost — is ~5× the cold tick at this
+shape (legacy kmeans lineage warm-start dominates; consistent with the
+smoke ratio). The runbook's 0.5-2 min/tick estimate was an order of
+magnitude optimistic — exactly why Julien required measurement. EC2
+r8g.4xlarge run in flight for the recorded number (expect same-or-worse
+per-core). Verdict drafting once EC2 lands, but the local number alone
+already says: NOT serial-OK at the extreme shape — the deterministic
+large-conv path (POST_CUTOVER_IMPROVEMENTS item 9) is REQUIRED before
+those 7 historical convs can be allowed to tick on Python, or they must
+be explicitly excluded at flip time.
+
+Independent review of #2659 (docs): every verifiable claim checked out;
+ONE finding (conf 85): the runbook Step-1 comment implied
+POLISMATH_ENGINE_MODE flows into the delphi-math-poller container, but
+docker-compose never wired it — an operator could have shadow-soaked in
+the wrong mode. ALREADY FIXED by this session's collapse commits (the
+env line is deleted from the runbook, compose, and example.env; the flag
+no longer exists). No action remaining; noted as validation that the
+collapse closed a real operational trap.
+
+#2660/#2661/#2662 are ISSUES (Clojure-bug documentation), not PRs — no
+diff to review (checked at Julien's request, s7).
+
+### Phase 5 DONE — EC2 measurement recorded (condition 6)
+
+r8g.4xlarge (i-057c881e212b2671d, self-terminated + verified), 33,422 x
+783, 2,005,320 synthesized votes: **cold tick 519.6s, warm tick
+1856.0s (~31 min)**. Local M-series cross-check 430.8s/2095.3s — same
+order; algorithmic, not instance-bound. Runbook risk item 3 updated with
+the numbers + verdict: NOT serial-OK at the extreme shape; item 9
+(deterministic large-conv path) or POLL_BLOCKLIST of the 7 historical
+zids required before they tick on Python; flip itself not blocked.
+Total EC2 cost: well under an hour of r8g.4xlarge (~$1).
+
+### Item-9 scope clarification (Julien question, s7)
+
+Q: does Clojure's large-conv special treatment change the k-means warm
+start? A (from source): NO — large-conv-update-graph merges
+small-conv-update-graph and overrides ONLY :pca (mini-batch PCA over an
+unseeded 1500-row twister sample, conversation.clj:760-773; sample-size
+line 745-757; dispatch cutoffs 10000/5000 at 785-796). :base-clusters
+and :group-clusterings (both warm-started) are inherited unchanged —
+Clojure ran the identical k-means machinery at 33k rows and got away
+with it on JVM/vectorz speed. Item 9 scope = deterministic sampled PCA
++ Python k-means performance (the warm tick's dominant cost), recorded
+in the runbook risk item 3.
+
+Final reviews: #2672/#2674/#2675 all SOUND (zero findings ≥80; the
+warm-tick methodology independently verified as genuinely steady-state
+via recompute()'s prev-state threading). #2663 sound with one pin
+applied (the _euclidean NaN test, above). All five python-ci dispatches
+this session: SUCCESS.
+
+## Session 7 FINAL: GOAL_CUTOVER_READY ACHIEVED — STATUS: DONE (2026-07-27)
+
+All seven conditions hold on the final tree (see GOAL_STATE.md for the
+condition-by-condition evidence and the walkthrough section). One
+session took the goal end-to-end: Phase 0 (battery speedup, A/B-proven),
+Phase 1 (inventory), Phase 2 (mode collapse, 7 PRs, battery 20/20 on the
+collapsed tree), Phase 3 (14b/14c, bit-identity proven), Phase 4
+(goldens verify-then-re-record, comparer 7/7, suite green, battery pair,
+equiv PASS ×2 live), Phase 5 (EC2 measurement + verdict + item-9 scope
+clarification). Every PR independently reviewed (all sound); five CI
+dispatches all green; EC2 instance terminated and verified; final
+battery pair re-run after the last docs edits: 20/20 MATCH ×2.
+
+### Julien ruling (s7, post-measurement): NO blocklisting; vectorize instead
+
+NO zid is ever blocklisted, and the k-means warm start STAYS (cluster-id
+stability between calls is user-facing). Item 9 re-scoped accordingly in
+POST_CUTOVER_IMPROVEMENTS.md + runbook risk item 3: (a) vectorize the
+warm-start k-means hot path — per-center BLAS distance columns
+(d2 = row_norms + |c|^2 - 2*(X@c), same cancellation formula) replacing
+~3.3M per-iteration python _euclidean calls at the 33k shape; expected
+10-100x on the dominant loop; ACCEPTANCE = bit-identity (Q11 0.0-ties
+decide merges/ids — pinned by the vw every-vote step-57 tie test and the
+full battery); (b) deterministic seeded sampled PCA for extreme shapes.
+Feasibility note: dgemv-per-center keeps each element a row-dot-center
+op (same class as the scalar np.dot), so tie reproduction is plausible;
+dgemm reassociation/FMA is the hazard to test for.

delphi/docs/CLOJURE_QUIRKS.md

@@ -29,7 +29,7 @@ Status legend: **REPLICATED** (Python legacy mode reproduces it) ·
 | Q8 | **Meta-tid `0`-is-truthy routing bug (#1961)** — every comment took the meta branch of `priority-metric`, flattening priorities to ~all-49. | math/.../repness or priorities path; see MATH_ALGORITHM_HISTORY.md | FIXED-UPSTREAM (#2611, merged 2026-07-18; not yet in a prod deploy as of 2026-07-21) | Done; watch prod deploy |
 | Q9 | **Subgroup smoother was UNCLAMPED in prod** until the #2575 clamp landed via #2609 (merged 2026-07-18). Only relevant to blobs generated by pre-#2609 workers. | conversation.clj:534-567 pre-#2609 | FIXED-UPSTREAM (and moot for us given Q7 carve-out) | — |
 | Q10 | **Large-conv mini-batch PCA is UNSEEDED-RANDOM** — `conv-update` dispatches to `large-conv-update` when n-ptpts > 10000 OR n-cmts > 5000 (conversation.clj:784-815); its `:pca` runs `partial-pca` on a fresh Mersenne-Twister row sample PER ITERATION with NO seed (`(sampling/sample (range n-ptpts) :generator :twister)`, conversation.clj:763-773) — so two Clojure runs of the same large conversation produce different PCA (and everything downstream). There is NO deterministic reference on this path, not even Clojure itself. Verified empirically 2026-07-22: pakistan/engage/bg2050 battery entries diverge at EXACTLY the first step crossing the cutoff (n-cmts 4914→6028, 4850→5765, 4694→6701) and match on every step before it. | conversation.clj:757-773 (sample-size + large graph), 784-815 (dispatch) | CARVED-OUT of R1 certification: the battery certifies the deterministic full-PCA path by pinning `{:ptpt-cutoff :cmt-cutoff}` HUGE in the replay drivers (conv-update accepts opts; math/dev/ change only — exclusion logged per run, never silent). Python keeps full PCA at all sizes. | Poller phase must decide the production-cutover story for large convs (python full-PCA is deterministic and strictly better; document as an intentional improvement over Clojure's randomized mini-batch). |
-| Q11 | **kmeans distances suffer dot-formula cancellation — near-coincident points TIE at exactly 0.0 and merge.** vectorz's `matrix/distance` on the row types kmeans actually passes (ArraySubVector view vs Vector center) computes d^2 = |a|^2+|b|^2-2ab, whose cancellation floors any true distance below ~1e-8 to EXACTLY 0.0 (verified in-process: pid-5/7 pair at true distance 4.66e-15 -> both cluster distances 0.0, while the same call with a copied row returns the true values). Ties then resolve via min-key LAST-wins -> near-coincident points collapse into the LATER cluster and the emptied cluster is dropped (vw every-vote step 57: cluster 8 absorbs [5 7], cluster 6 dropped). Python's norm(a-b) has no cancellation -> no tie -> no merge. | clusters.clj:44-52 (add-to-closest via matrix/distance); verified 2026-07-22 via math/dev/proj_probe.clj | TO REPLICATE (legacy mode): compute kmeans distances as sqrt(max(0, |a|^2+|b|^2-2ab)) in float64 — reproduces the 0.0 tie deterministically (numpy check confirms) | CORRECTED 2026-07-22 s4: `most-distal` does NOT use the cancellation formula — its rows come from `get-row-by-name` (matrix/get-row), a type on which `matrix/distance` returns the TRUE value (pc-revote-01 probe: clj's near-tie gaps are 2.5e-16/5.6e-17 = true-formula-sized, not the 1e-14 the cancellation formula produces on the same points). Python's port applies the cancellation formula at the shared helper, so py most_distal differs from clj in the low bits — benign wherever gaps exceed ~1e-8·scale; the only regime where it matters is the Q13 knife-edge class, which is carved out. Left as-is deliberately. |
+| Q11 | **kmeans distances suffer dot-formula cancellation — near-coincident points TIE at exactly 0.0 and merge.** vectorz's `matrix/distance` on the row types kmeans actually passes (ArraySubVector view vs Vector center) computes d^2 = |a|^2+|b|^2-2ab, whose cancellation floors any true distance below ~1e-8 to EXACTLY 0.0 (verified in-process: pid-5/7 pair at true distance 4.66e-15 -> both cluster distances 0.0, while the same call with a copied row returns the true values). Ties then resolve via min-key LAST-wins -> near-coincident points collapse into the LATER cluster and the emptied cluster is dropped (vw every-vote step 57: cluster 8 absorbs [5 7], cluster 6 dropped). Python's norm(a-b) has no cancellation -> no tie -> no merge. | clusters.clj:44-52 (add-to-closest via matrix/distance); verified 2026-07-22 via math/dev/proj_probe.clj | TO REPLICATE (legacy mode): compute kmeans distances as sqrt(max(0, |a|^2+|b|^2-2ab)) in float64 — reproduces the 0.0 tie deterministically (numpy check confirms) | CORRECTED 2026-07-22 s4: `most-distal` does NOT use the cancellation formula — its rows come from `get-row-by-name` (matrix/get-row), a type on which `matrix/distance` returns the TRUE value (pc-revote-01 probe: clj's near-tie gaps are 2.5e-16/5.6e-17 = true-formula-sized, not the 1e-14 the cancellation formula produces on the same points). Python's port applies the cancellation formula at the shared helper, so py most_distal differs from clj in the low bits — benign wherever gaps exceed ~1e-8·scale; the only regime where it matters is the Q13 knife-edge class, which is carved out. Left as-is deliberately. NaN addendum (s7, #2663 review): the port lets NaN PROPAGATE through _euclidean (real vectorz has no clamp); only negative cancellation residue is floored to 0.0 — pinned by test_euclidean_propagates_nan_instead_of_clamping. |
 | Q12 | **Cold-tick PCA start vector is UNSEEDED-RANDOM** (`rand-starting-vec`, pca.clj:79-82 — the original author's own 'should really throw a [seeded] random number generator in the equation here... XXX' comment). With a small eigengap the fixed 100 power iterations do NOT fully converge, so the start-dependent residual (~1e-4 on pc-smallmix-01 step 0) survives into comps/projections — even two Clojure runs differ on the cold tick. Warm ticks are unaffected (start-vectors = previous comps). | pca.clj:79-101 | CARVED OUT for certification: BOTH replay drivers pin the cold start to the ONES vector — the exact value power-iteration already pads new-comment columns with (pca.clj:46-49 / pca.py _power_iteration) — via a single-element [1.0] start that padding expands to any width (dev/replay.clj certify-cold-start-pca + replay/driver.py seed). Logged per run. | Production: seed the start vector (the author's own XXX) — deterministic cold ticks with no behavior change at convergence |
 | Q13 | **Warm-chain split-loop extraction order is knife-edge-chaotic on tie-dense geometry.** `clean-start-clusters`' split loop (clusters.clj:250-273) extracts the most-distal point one at a time; when several near-coincident candidates tie (within-engine distance gaps at the few-ulp level, e.g. 2.5e-16/5.6e-17 on pc-revote-01 step 1 among 4 pids), the extraction ORDER — and thus the minted singleton ids and final partition — is decided by sub-ulp arithmetic noise. Cross-engine, projections differ at ~1e-5 (residual small-eigengap power-iteration noise, tolerant-accepted), ELEVEN orders above the gaps: no arithmetic replication can reproduce Clojure's order (verified 2026-07-22: both probes extract the identical 28-pid sequence, then clj picks {82,99} where py picks {80,99} from the tied set {80,82,99,108}; even the true-distance formula ranks them differently per engine). Same irreducibility class as the vw every-vote step-57 knife edge. | clusters.clj:202-217 (most-distal), 250-273 (split loop); probes: math/dev/proj_probe.clj split-probe + delphi/scratch/probe_revote_split.py | CARVED OUT: battery keeps revote coverage via pc-revote-02 (34 ptpts, ~28% revotes — rich geometry, no knife edge; MATCH 6/6 first try) in place of pc-revote-01 (205 ptpts on 15 comments — near-discrete projection space). Ledgered on FP-912391ece7/FP-c29173e1ba/FP-98dc728043. | Not a Clojure bug — an irreducible float-chaos regime. Any future dataset that diverges ONLY in split-loop extraction order on few-ulp gaps belongs to this class: probe with split-probe, then swap or document. |
 | Q14 | **No small-dimension guards — tiny matrices run the real math.** Clojure runs powerit PCA on 1x1/1xN/Nx1 rating matrices (real center, comps rank-capped at min(rows,cols)) and `conv-repness` always returns its best-agree comment even for degenerate shapes — where pre-parity Python short-circuited: PCA returned zeros (center -0.0 vs clj -1.0 on a 1x1 matrix; comps zero-padded to 2 vs rank-capped 1) and `conv_repness` returned empty repness/consensus below shape 2. | pca.clj (powerit-pca, no dim guard); repness.clj conv-repness (best-agree fallback) | REPLICATED (2026-07-22, PR #2653: legacy mode relaxes the `<2` guards in conversation.py::_compute_pca, pca.py::pca_project_dataframe, repness.py::conv_repness; single-vote fixtures with exact Clojure-derived values, tests/test_legacy_blob_shape.py; certified end-to-end by the vw-every-vote-56 battery entry whose early steps are 1xN). Ledgered late — flagged by the 2026-07-22 s4 review pass; row added then. | `improved` mode keeps the guards (sane early returns) |

delphi/docs/CUTOVER_RUNBOOK.md

@@ -28,11 +28,36 @@ equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19).
    acceptance for them up front (structural-only, or exclude from compare).
 3. **Poller throughput**: ~1.66 ticks/s per process — measured on EC2
    (r8g.4xlarge, cost-model study) on biodiversity-sized replays.
-   **PRE-FLIP MEASUREMENT REQUIRED: one full-PCA tick of the largest
-   prodclone conv (33k ptpts) on the target EC2 instance** — 7 historical
-   convs sit above the old Clojure large-conv cutoffs and python runs
-   full PCA at every size (est. 0.5-2 min/tick at the extreme; fine for
-   7 rarely-active convs, but measure, don't estimate).
+   **PRE-FLIP MEASUREMENT DONE (2026-07-27, s7)** — one full-PCA tick of
+   the largest prodclone shape (33,422 ptpts x 783 cmts, 2.0M votes;
+   synthesized, seeded) on r8g.4xlarge via
+   scripts/large_conv_tick_bench.py:
+   **cold tick 519.6s (~8.7 min); WARM (steady-state) tick 1856.0s
+   (~30.9 min)** — the warm tick is ~3.6x the cold one (legacy kmeans
+   lineage warm-start dominates). Local M-series cross-check: 430.8s /
+   2095.3s — same order, so this is algorithmic, not instance-bound.
+   **VERDICT: serial is NOT OK at the extreme shape** — the old 0.5-2
+   min/tick estimate was an order of magnitude optimistic. A ~31-minute
+   tick would occupy a poller process/shard for its duration whenever one
+   of the 7 historical large convs receives votes. REQUIRED fix
+   (Julien ruling, s7: NO zid is ever blocklisted, and the warm start
+   STAYS — cluster-id stability across ticks is user-facing): item 9,
+   re-scoped as (a) VECTORIZE the warm-start k-means hot path — replace
+   the per-pair python _euclidean loop with per-center BLAS columns
+   (d2 = row_norms + |c|^2 - 2*(X@c), the same cancellation formula) in
+   cluster_step/most_distal/weighted_mean; bit-identity is the
+   acceptance bar (Q11 0.0-ties are load-bearing for cluster ids — the
+   vw every-vote step-57 tie test + the full battery gate it); plus
+   (b) a deterministic (seeded) sampled PCA for the extreme shapes.
+   SCOPE NOTE (Julien question, s7): Clojure's large-conv graph
+   overrides ONLY the :pca node (mini-batch PCA over an unseeded
+   1500-row sample; conversation.clj:760-773 — large-conv-update-graph
+   merges small-conv-update-graph) — the k-means warm start is IDENTICAL
+   in both graphs, so there is no Clojure-side large-conv k-means
+   treatment to port; vectorz's JVM loops simply outran our per-cluster
+   Python port at 33k rows. The flip is NOT blocked: all 7 large convs
+   are historical and rarely active; if one ticks before item 9 lands it
+   is slow (~31 min) but correct and stable.
    Sharding (#2658) is the scale-out path, opt-in via POLL_SHARD_INDEX/
    POLL_SHARD_COUNT — one shard = one process. Start UNSHARDED (defaults
    are a verified no-op); shard only if the shadow soak shows lag.