diff --git a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md index 7b09f29c2..13d0b2b80 100644 --- a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md +++ b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md @@ -4288,3 +4288,126 @@ PR #2673 review agent verdict: CLEAN — split proven behavior-preserving (including an equivalence proof of the counts_df.empty seam), blob test non-vacuous (45+42 entries), tolerances justified against Clojure's `(float repness-test)` cast at repness.clj:187. + +### Conditions 4+5 EVIDENCE (s7 cont.) + +- **Battery pair** (condition 4): 20/20 MATCH ×2 consecutive on the + Phase-4 tree; divergences.json = 81 entries, 0 open. +- **Equivalence release gate** (condition 5): poller_equiv.py full-run + re-run LIVE on the collapsed+refactored tree — + vw: verdict PASS/MATCH, 8/8 batches, ticks OK both envs, 0 + envelope-excused divergences (worst self-jitter 4.9e-06); + pc-meta-02: verdict PASS/MATCH, 8/8 batches, 0 envelope-excused + (worst self-jitter 1.5e-06). Evidence refreshed in place under + real_data/.local/replays/poller_equiv/{vw,pc-meta-02}/. +- CI: run 30286254481 (collapse tip) SUCCESS; run 30288678922 (#2673) + SUCCESS; run 30302469207 (#2675 tip) dispatched. +- PRs #2674 (FLI xfail + journal) + #2675 (large-conv bench tool) + pushed. Public golden_snapshot.json now gitignored (goldens stay + local artifacts; a 350k-line accidental snapshot was stripped from + #2674 before push). + +### Phase 5 IN FLIGHT (s7 cont.) + +EC2 measurement launched: i-057c881e212b2671d (r8g.4xlarge, us-east-1, +bench profile, Project=polis-cost-model), self-terminating user-data +(clone spr/edge/5747f8c9 → minimal venv (numpy/pandas/sklearn/natsort/ +click/pyyaml) → scripts/large_conv_tick_bench.py full 33k×783 shape → +S3 results/large-conv-tick/ → SQS polis-cost-model-done → shutdown; +dead-man 180 min; terminate-on-shutdown). Local full-size run in +parallel on the M-series laptop for a comparison point. Smoke numbers +(2000×200×120k, laptop): cold 6.5s, warm 14.4s — warm is the expensive +side (legacy kmeans warm-start path dominates). + +### Review protocol correction (Julien, s7): Copilot credits exhausted + +The Copilot reviews requested on #2659/#2663 never ran — monthly AI +credits are exhausted again. Per Julien: use INDEPENDENT /code-review +subagents instead. Two launched (one per PR); do not re-request Copilot +this month. (The collapse series #2665-#2671 and #2673 already had +independent review-agent passes — both CLEAN.) + +### Phase 5 — LOCAL full-size result + #2659 review disposition (s7 cont.) + +Local (M-series laptop, arm64), 33,422 × 783, 2,005,320 votes: +ingest 2.4s, **cold tick 430.8s (~7.2 min), WARM tick 2095.3s (~35 min)**. +The warm tick — the steady per-tick cost — is ~5× the cold tick at this +shape (legacy kmeans lineage warm-start dominates; consistent with the +smoke ratio). The runbook's 0.5-2 min/tick estimate was an order of +magnitude optimistic — exactly why Julien required measurement. EC2 +r8g.4xlarge run in flight for the recorded number (expect same-or-worse +per-core). Verdict drafting once EC2 lands, but the local number alone +already says: NOT serial-OK at the extreme shape — the deterministic +large-conv path (POST_CUTOVER_IMPROVEMENTS item 9) is REQUIRED before +those 7 historical convs can be allowed to tick on Python, or they must +be explicitly excluded at flip time. + +Independent review of #2659 (docs): every verifiable claim checked out; +ONE finding (conf 85): the runbook Step-1 comment implied +POLISMATH_ENGINE_MODE flows into the delphi-math-poller container, but +docker-compose never wired it — an operator could have shadow-soaked in +the wrong mode. ALREADY FIXED by this session's collapse commits (the +env line is deleted from the runbook, compose, and example.env; the flag +no longer exists). No action remaining; noted as validation that the +collapse closed a real operational trap. + +#2660/#2661/#2662 are ISSUES (Clojure-bug documentation), not PRs — no +diff to review (checked at Julien's request, s7). + +### Phase 5 DONE — EC2 measurement recorded (condition 6) + +r8g.4xlarge (i-057c881e212b2671d, self-terminated + verified), 33,422 x +783, 2,005,320 synthesized votes: **cold tick 519.6s, warm tick +1856.0s (~31 min)**. Local M-series cross-check 430.8s/2095.3s — same +order; algorithmic, not instance-bound. Runbook risk item 3 updated with +the numbers + verdict: NOT serial-OK at the extreme shape; item 9 +(deterministic large-conv path) or POLL_BLOCKLIST of the 7 historical +zids required before they tick on Python; flip itself not blocked. +Total EC2 cost: well under an hour of r8g.4xlarge (~$1). + +### Item-9 scope clarification (Julien question, s7) + +Q: does Clojure's large-conv special treatment change the k-means warm +start? A (from source): NO — large-conv-update-graph merges +small-conv-update-graph and overrides ONLY :pca (mini-batch PCA over an +unseeded 1500-row twister sample, conversation.clj:760-773; sample-size +line 745-757; dispatch cutoffs 10000/5000 at 785-796). :base-clusters +and :group-clusterings (both warm-started) are inherited unchanged — +Clojure ran the identical k-means machinery at 33k rows and got away +with it on JVM/vectorz speed. Item 9 scope = deterministic sampled PCA ++ Python k-means performance (the warm tick's dominant cost), recorded +in the runbook risk item 3. + +Final reviews: #2672/#2674/#2675 all SOUND (zero findings ≥80; the +warm-tick methodology independently verified as genuinely steady-state +via recompute()'s prev-state threading). #2663 sound with one pin +applied (the _euclidean NaN test, above). All five python-ci dispatches +this session: SUCCESS. + +## Session 7 FINAL: GOAL_CUTOVER_READY ACHIEVED — STATUS: DONE (2026-07-27) + +All seven conditions hold on the final tree (see GOAL_STATE.md for the +condition-by-condition evidence and the walkthrough section). One +session took the goal end-to-end: Phase 0 (battery speedup, A/B-proven), +Phase 1 (inventory), Phase 2 (mode collapse, 7 PRs, battery 20/20 on the +collapsed tree), Phase 3 (14b/14c, bit-identity proven), Phase 4 +(goldens verify-then-re-record, comparer 7/7, suite green, battery pair, +equiv PASS ×2 live), Phase 5 (EC2 measurement + verdict + item-9 scope +clarification). Every PR independently reviewed (all sound); five CI +dispatches all green; EC2 instance terminated and verified; final +battery pair re-run after the last docs edits: 20/20 MATCH ×2. + +### Julien ruling (s7, post-measurement): NO blocklisting; vectorize instead + +NO zid is ever blocklisted, and the k-means warm start STAYS (cluster-id +stability between calls is user-facing). Item 9 re-scoped accordingly in +POST_CUTOVER_IMPROVEMENTS.md + runbook risk item 3: (a) vectorize the +warm-start k-means hot path — per-center BLAS distance columns +(d2 = row_norms + |c|^2 - 2*(X@c), same cancellation formula) replacing +~3.3M per-iteration python _euclidean calls at the 33k shape; expected +10-100x on the dominant loop; ACCEPTANCE = bit-identity (Q11 0.0-ties +decide merges/ids — pinned by the vw every-vote step-57 tie test and the +full battery); (b) deterministic seeded sampled PCA for extreme shapes. +Feasibility note: dgemv-per-center keeps each element a row-dot-center +op (same class as the scalar np.dot), so tie reproduction is plausible; +dgemm reassociation/FMA is the hazard to test for. diff --git a/delphi/docs/CLOJURE_QUIRKS.md b/delphi/docs/CLOJURE_QUIRKS.md index 75fe1663f..ba565e56b 100644 --- a/delphi/docs/CLOJURE_QUIRKS.md +++ b/delphi/docs/CLOJURE_QUIRKS.md @@ -29,7 +29,7 @@ Status legend: **REPLICATED** (Python legacy mode reproduces it) · | Q8 | **Meta-tid `0`-is-truthy routing bug (#1961)** — every comment took the meta branch of `priority-metric`, flattening priorities to ~all-49. | math/.../repness or priorities path; see MATH_ALGORITHM_HISTORY.md | FIXED-UPSTREAM (#2611, merged 2026-07-18; not yet in a prod deploy as of 2026-07-21) | Done; watch prod deploy | | Q9 | **Subgroup smoother was UNCLAMPED in prod** until the #2575 clamp landed via #2609 (merged 2026-07-18). Only relevant to blobs generated by pre-#2609 workers. | conversation.clj:534-567 pre-#2609 | FIXED-UPSTREAM (and moot for us given Q7 carve-out) | — | | Q10 | **Large-conv mini-batch PCA is UNSEEDED-RANDOM** — `conv-update` dispatches to `large-conv-update` when n-ptpts > 10000 OR n-cmts > 5000 (conversation.clj:784-815); its `:pca` runs `partial-pca` on a fresh Mersenne-Twister row sample PER ITERATION with NO seed (`(sampling/sample (range n-ptpts) :generator :twister)`, conversation.clj:763-773) — so two Clojure runs of the same large conversation produce different PCA (and everything downstream). There is NO deterministic reference on this path, not even Clojure itself. Verified empirically 2026-07-22: pakistan/engage/bg2050 battery entries diverge at EXACTLY the first step crossing the cutoff (n-cmts 4914→6028, 4850→5765, 4694→6701) and match on every step before it. | conversation.clj:757-773 (sample-size + large graph), 784-815 (dispatch) | CARVED-OUT of R1 certification: the battery certifies the deterministic full-PCA path by pinning `{:ptpt-cutoff :cmt-cutoff}` HUGE in the replay drivers (conv-update accepts opts; math/dev/ change only — exclusion logged per run, never silent). Python keeps full PCA at all sizes. | Poller phase must decide the production-cutover story for large convs (python full-PCA is deterministic and strictly better; document as an intentional improvement over Clojure's randomized mini-batch). | -| Q11 | **kmeans distances suffer dot-formula cancellation — near-coincident points TIE at exactly 0.0 and merge.** vectorz's `matrix/distance` on the row types kmeans actually passes (ArraySubVector view vs Vector center) computes d^2 = |a|^2+|b|^2-2ab, whose cancellation floors any true distance below ~1e-8 to EXACTLY 0.0 (verified in-process: pid-5/7 pair at true distance 4.66e-15 -> both cluster distances 0.0, while the same call with a copied row returns the true values). Ties then resolve via min-key LAST-wins -> near-coincident points collapse into the LATER cluster and the emptied cluster is dropped (vw every-vote step 57: cluster 8 absorbs [5 7], cluster 6 dropped). Python's norm(a-b) has no cancellation -> no tie -> no merge. | clusters.clj:44-52 (add-to-closest via matrix/distance); verified 2026-07-22 via math/dev/proj_probe.clj | TO REPLICATE (legacy mode): compute kmeans distances as sqrt(max(0, |a|^2+|b|^2-2ab)) in float64 — reproduces the 0.0 tie deterministically (numpy check confirms) | CORRECTED 2026-07-22 s4: `most-distal` does NOT use the cancellation formula — its rows come from `get-row-by-name` (matrix/get-row), a type on which `matrix/distance` returns the TRUE value (pc-revote-01 probe: clj's near-tie gaps are 2.5e-16/5.6e-17 = true-formula-sized, not the 1e-14 the cancellation formula produces on the same points). Python's port applies the cancellation formula at the shared helper, so py most_distal differs from clj in the low bits — benign wherever gaps exceed ~1e-8·scale; the only regime where it matters is the Q13 knife-edge class, which is carved out. Left as-is deliberately. | +| Q11 | **kmeans distances suffer dot-formula cancellation — near-coincident points TIE at exactly 0.0 and merge.** vectorz's `matrix/distance` on the row types kmeans actually passes (ArraySubVector view vs Vector center) computes d^2 = |a|^2+|b|^2-2ab, whose cancellation floors any true distance below ~1e-8 to EXACTLY 0.0 (verified in-process: pid-5/7 pair at true distance 4.66e-15 -> both cluster distances 0.0, while the same call with a copied row returns the true values). Ties then resolve via min-key LAST-wins -> near-coincident points collapse into the LATER cluster and the emptied cluster is dropped (vw every-vote step 57: cluster 8 absorbs [5 7], cluster 6 dropped). Python's norm(a-b) has no cancellation -> no tie -> no merge. | clusters.clj:44-52 (add-to-closest via matrix/distance); verified 2026-07-22 via math/dev/proj_probe.clj | TO REPLICATE (legacy mode): compute kmeans distances as sqrt(max(0, |a|^2+|b|^2-2ab)) in float64 — reproduces the 0.0 tie deterministically (numpy check confirms) | CORRECTED 2026-07-22 s4: `most-distal` does NOT use the cancellation formula — its rows come from `get-row-by-name` (matrix/get-row), a type on which `matrix/distance` returns the TRUE value (pc-revote-01 probe: clj's near-tie gaps are 2.5e-16/5.6e-17 = true-formula-sized, not the 1e-14 the cancellation formula produces on the same points). Python's port applies the cancellation formula at the shared helper, so py most_distal differs from clj in the low bits — benign wherever gaps exceed ~1e-8·scale; the only regime where it matters is the Q13 knife-edge class, which is carved out. Left as-is deliberately. NaN addendum (s7, #2663 review): the port lets NaN PROPAGATE through _euclidean (real vectorz has no clamp); only negative cancellation residue is floored to 0.0 — pinned by test_euclidean_propagates_nan_instead_of_clamping. | | Q12 | **Cold-tick PCA start vector is UNSEEDED-RANDOM** (`rand-starting-vec`, pca.clj:79-82 — the original author's own 'should really throw a [seeded] random number generator in the equation here... XXX' comment). With a small eigengap the fixed 100 power iterations do NOT fully converge, so the start-dependent residual (~1e-4 on pc-smallmix-01 step 0) survives into comps/projections — even two Clojure runs differ on the cold tick. Warm ticks are unaffected (start-vectors = previous comps). | pca.clj:79-101 | CARVED OUT for certification: BOTH replay drivers pin the cold start to the ONES vector — the exact value power-iteration already pads new-comment columns with (pca.clj:46-49 / pca.py _power_iteration) — via a single-element [1.0] start that padding expands to any width (dev/replay.clj certify-cold-start-pca + replay/driver.py seed). Logged per run. | Production: seed the start vector (the author's own XXX) — deterministic cold ticks with no behavior change at convergence | | Q13 | **Warm-chain split-loop extraction order is knife-edge-chaotic on tie-dense geometry.** `clean-start-clusters`' split loop (clusters.clj:250-273) extracts the most-distal point one at a time; when several near-coincident candidates tie (within-engine distance gaps at the few-ulp level, e.g. 2.5e-16/5.6e-17 on pc-revote-01 step 1 among 4 pids), the extraction ORDER — and thus the minted singleton ids and final partition — is decided by sub-ulp arithmetic noise. Cross-engine, projections differ at ~1e-5 (residual small-eigengap power-iteration noise, tolerant-accepted), ELEVEN orders above the gaps: no arithmetic replication can reproduce Clojure's order (verified 2026-07-22: both probes extract the identical 28-pid sequence, then clj picks {82,99} where py picks {80,99} from the tied set {80,82,99,108}; even the true-distance formula ranks them differently per engine). Same irreducibility class as the vw every-vote step-57 knife edge. | clusters.clj:202-217 (most-distal), 250-273 (split loop); probes: math/dev/proj_probe.clj split-probe + delphi/scratch/probe_revote_split.py | CARVED OUT: battery keeps revote coverage via pc-revote-02 (34 ptpts, ~28% revotes — rich geometry, no knife edge; MATCH 6/6 first try) in place of pc-revote-01 (205 ptpts on 15 comments — near-discrete projection space). Ledgered on FP-912391ece7/FP-c29173e1ba/FP-98dc728043. | Not a Clojure bug — an irreducible float-chaos regime. Any future dataset that diverges ONLY in split-loop extraction order on few-ulp gaps belongs to this class: probe with split-probe, then swap or document. | | Q14 | **No small-dimension guards — tiny matrices run the real math.** Clojure runs powerit PCA on 1x1/1xN/Nx1 rating matrices (real center, comps rank-capped at min(rows,cols)) and `conv-repness` always returns its best-agree comment even for degenerate shapes — where pre-parity Python short-circuited: PCA returned zeros (center -0.0 vs clj -1.0 on a 1x1 matrix; comps zero-padded to 2 vs rank-capped 1) and `conv_repness` returned empty repness/consensus below shape 2. | pca.clj (powerit-pca, no dim guard); repness.clj conv-repness (best-agree fallback) | REPLICATED (2026-07-22, PR #2653: legacy mode relaxes the `<2` guards in conversation.py::_compute_pca, pca.py::pca_project_dataframe, repness.py::conv_repness; single-vote fixtures with exact Clojure-derived values, tests/test_legacy_blob_shape.py; certified end-to-end by the vw-every-vote-56 battery entry whose early steps are 1xN). Ledgered late — flagged by the 2026-07-22 s4 review pass; row added then. | `improved` mode keeps the guards (sane early returns) | diff --git a/delphi/docs/CUTOVER_RUNBOOK.md b/delphi/docs/CUTOVER_RUNBOOK.md index 8f3ea4750..0ab51a878 100644 --- a/delphi/docs/CUTOVER_RUNBOOK.md +++ b/delphi/docs/CUTOVER_RUNBOOK.md @@ -28,11 +28,36 @@ equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19). acceptance for them up front (structural-only, or exclude from compare). 3. **Poller throughput**: ~1.66 ticks/s per process — measured on EC2 (r8g.4xlarge, cost-model study) on biodiversity-sized replays. - **PRE-FLIP MEASUREMENT REQUIRED: one full-PCA tick of the largest - prodclone conv (33k ptpts) on the target EC2 instance** — 7 historical - convs sit above the old Clojure large-conv cutoffs and python runs - full PCA at every size (est. 0.5-2 min/tick at the extreme; fine for - 7 rarely-active convs, but measure, don't estimate). + **PRE-FLIP MEASUREMENT DONE (2026-07-27, s7)** — one full-PCA tick of + the largest prodclone shape (33,422 ptpts x 783 cmts, 2.0M votes; + synthesized, seeded) on r8g.4xlarge via + scripts/large_conv_tick_bench.py: + **cold tick 519.6s (~8.7 min); WARM (steady-state) tick 1856.0s + (~30.9 min)** — the warm tick is ~3.6x the cold one (legacy kmeans + lineage warm-start dominates). Local M-series cross-check: 430.8s / + 2095.3s — same order, so this is algorithmic, not instance-bound. + **VERDICT: serial is NOT OK at the extreme shape** — the old 0.5-2 + min/tick estimate was an order of magnitude optimistic. A ~31-minute + tick would occupy a poller process/shard for its duration whenever one + of the 7 historical large convs receives votes. REQUIRED fix + (Julien ruling, s7: NO zid is ever blocklisted, and the warm start + STAYS — cluster-id stability across ticks is user-facing): item 9, + re-scoped as (a) VECTORIZE the warm-start k-means hot path — replace + the per-pair python _euclidean loop with per-center BLAS columns + (d2 = row_norms + |c|^2 - 2*(X@c), the same cancellation formula) in + cluster_step/most_distal/weighted_mean; bit-identity is the + acceptance bar (Q11 0.0-ties are load-bearing for cluster ids — the + vw every-vote step-57 tie test + the full battery gate it); plus + (b) a deterministic (seeded) sampled PCA for the extreme shapes. + SCOPE NOTE (Julien question, s7): Clojure's large-conv graph + overrides ONLY the :pca node (mini-batch PCA over an unseeded + 1500-row sample; conversation.clj:760-773 — large-conv-update-graph + merges small-conv-update-graph) — the k-means warm start is IDENTICAL + in both graphs, so there is no Clojure-side large-conv k-means + treatment to port; vectorz's JVM loops simply outran our per-cluster + Python port at 33k rows. The flip is NOT blocked: all 7 large convs + are historical and rarely active; if one ticks before item 9 lands it + is slow (~31 min) but correct and stable. Sharding (#2658) is the scale-out path, opt-in via POLL_SHARD_INDEX/ POLL_SHARD_COUNT — one shard = one process. Start UNSHARDED (defaults are a verified no-op); shard only if the shadow soak shows lag. diff --git a/delphi/docs/GOAL_STATE.md b/delphi/docs/GOAL_STATE.md index d9dfe1976..8a291e9d1 100644 --- a/delphi/docs/GOAL_STATE.md +++ b/delphi/docs/GOAL_STATE.md @@ -1,54 +1,53 @@ -STATUS: IN PROGRESS +STATUS: DONE -# GOAL_STATE — checkpoint for GOAL_CUTOVER_READY.md (cap ~50 lines) +# GOAL_STATE — GOAL_CUTOVER_READY.md ACHIEVED (2026-07-27, session 7) -## Where we are (2026-07-27, session 7 wind-down) +All seven DONE conditions hold, evidenced in-repo: -- Phase 0 DONE + A/B-proven: engine-scoped py cache hash + 6-way parallel - battery (PR #2664, python-ci green, review clean). Costs now: no edit - ~22s, harness-only edit ~2m, engine edit ~19m (long pole - pakistan:uniform8 ~18m — <8m target unreachable without intra-entry - parallelism; journal s7). -- Phase 1 DONE: engine_mode inventory journaled (s7). -- Phase 2 DONE — MODE COLLAPSE EXECUTED: 7 commits = PRs #2665-#2671. - DONE-gate grep = 0 hits over delphi/polismath/. Ban filtering DELETED - outright. Parks minted: jj bookmarks improvements/item-{2,4,5,8} - (verbatim reverse patches off the C7 commit; NOT buildable — they - carry old flag refs; re-landing = keep the improved side only). - Battery on the collapsed tree: 20/20 MATCH full py re-replay (19m21s) - + cached pass 22s. Full suite 1155 passed / 22 skipped / 44 xfailed - / 2 XPASS (D9/D10 vw-cold_start now match Clojure — parity improved). -- Phase 3 DONE: 14b blob-injection pins + 14c two-phase split (PR - #2673); battery 20/20 on the refactored tree (bit-identity); collapse - review agent verdict CLEAN, its 3 cleanups applied. #2664 CI green. +1. Mode collapse: gate grep (ENGINE_MODE|engine_mode|resolve_engine_mode + over delphi/polismath/) = 0 hits; ban filtering DELETED outright; + parks = jj bookmarks improvements/item-{2,4,5,8} (pushed to origin; + verbatim reverse patches, NOT buildable — re-land = keep improved + side). PRs #2665-#2671. run_delphi/job_poller share the single path. +2. Clarity refactor 14b/14c landed: PR #2673 (two-phase split + + blob-injection pins), battery-proven bit-identical. +3. Goldens re-recorded at the collapse tree (verify-then-record, journal + s7): comparer 7/7 PASS; --include-local suite green (one justified + Q12 xfail on FLI-cold_start prod-blob comparison, #2674). +4. Battery: 20/20 MATCH ×2 consecutive on the final tree (re-run after + the last docs edits); divergences.json 81 entries, 0 open. +5. Equivalence release gate: poller_equiv full-run PASS live on the + final code tree — vw 8/8 MATCH, pc-meta-02 8/8 MATCH, non-vacuous, + 0 envelope-excused divergences. +6. EC2 measurement in CUTOVER_RUNBOOK risk item 3: r8g.4xlarge, + 33,422×783/2.0M votes — cold 519.6s, WARM 1856.0s (~31 min). + VERDICT: NOT serial-OK at the extreme shape; fix = item 9 re-scoped + (vectorize warm-start k-means, bit-identity-gated + seeded sampled + PCA; NO blocklisting — Julien ruling s7); flip not blocked (slow but + correct+stable if a large conv ticks first). +7. This line 1 flip. -## Next actions +## For walkthrough (Julien) -1. Orientation: check CI runs 30286254481 (collapse tip) + 30288678922 - (#2673 14b/c) + the #2673 review agent result; Copilot comments on - #2659/#2663. Triage per protocol (a "fix" undoing legacy semantics = - post-cutover queue item). Phase 3 is DONE (PR #2673; battery 20/20 on - the refactored tree = bit-identity; journal s7 cont.). -2. Phase 4: goldens. Diagnostic done (journal s7 cont.): public datasets - have NO goldens (comparisons skip); the 5 private goldens are at - real_data/.local/*/golden_snapshot.json. Verify current private-golden - drift with `uv run python scripts/regression_comparer.py --include-local` - (needs /private-data setup), cross-check intended new values against - the battery clj recordings, THEN re-record via - scripts/regression_recorder.py; full suite --include-local green. -3. Phase 4 gates on the final tree: TWO consecutive clean battery passes - + poller_equiv.py full-run PASS (non-vacuous) on vw AND pc-meta-02 - (pgproxy 127.0.0.1:15432 up: `docker start polis-dev-postgres-1 - pgproxy`; docker compose services for the clj container). -4. Phase 5: EC2 large-conv tick measurement (bench AWS profile, 33k - ptpts × 783 cmts shape) → number + serial-OK verdict into - CUTOVER_RUNBOOK.md risk register. -5. Only after all of 1-6 of the DONE list hold on the final tree: flip - line 1 here to STATUS: DONE. +- Collapse series #2665-#2671 + #2673 + #2674 + #2675 (+docs #2672, + #2659, triage #2663, Phase-0 #2664). ALL independently review-agent'd: + clean (one pin applied: _euclidean NaN test, #2663 review). All 5 + python-ci dispatches green. Copilot NOT used (credits exhausted — + independent agents instead, Julien ruling s7). +- Battery cost model now: no edit ~22s / harness edit ~2m / engine edit + ~19m (pakistan long pole). Phase 0 A/B data in journal s7. +- Big deduced finding: warm tick ≫ cold tick at scale (legacy kmeans + warm start); Clojure's large-conv path never special-cased kmeans + (only :pca — conversation.clj:760-773), so item 9 = seeded sampled + PCA + kmeans performance. Runbook risk item 3 has the numbers. +- Next goal candidates: execute CUTOVER_RUNBOOK steps 0-3 (land, shadow, + flip, decommission — needs Julien/team); post-cutover queue items + (POST_CUTOVER_IMPROVEMENTS.md), item 9 first if large convs matter. ## Pointers - Contract: GOAL_CUTOVER_READY.md. Roadmap: POST_CUTOVER_IMPROVEMENTS.md. +- Runbook: CUTOVER_RUNBOOK.md. Journal: CLJ-PARITY-FIXES-JOURNAL.md s7. - Battery: cd delphi && uv run python scripts/certify.py run (workers=6). -- Suite baseline: 1155/22/44 + 2 xpassed (s7 collapse tree). -- s7 journal: Phase 0/1/2 + battery evidence (CLJ-PARITY-FIXES-JOURNAL.md). +- Suite baseline: 1155/22/44 (+2 xpassed) without --include-local; 1285+1xfail with + --include-local. diff --git a/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md b/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md index 2d16ba1a3..b121d0c5f 100644 --- a/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md +++ b/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md @@ -75,15 +75,18 @@ items. 8. Modern solver paths (sklearn PCA/k-means) where they beat the ports — the original "improved" aspiration, now landing with certification discipline. -9. **Deterministic large-conversation handling** (Julien 2026-07-27, from - the Q10 discussion): 7 of 15,575 prodclone convs ever crossed the - Clojure cutoffs (largest: 33,422 ptpts x 783 cmts / 2.0M votes; - 18,082 x 9,030). Python currently runs full PCA at every size — - deterministic but O(ptpts x cmts x iters) per tick; large-conv tick - cost on EC2 is UNMEASURED (see CUTOVER_RUNBOOK risk register). If - measurement says it is too slow, add a DETERMINISTIC large-conv path - (seeded mini-batch or randomized-SVD with fixed seed) as its own PR — - keeping determinism, unlike Clojure's unseeded sampling (#2660). +9. **Large-conversation performance, warm start preserved** (re-scoped + by Julien s7 after the EC2 measurement: r8g.4xlarge warm tick + 1856s at 33,422 x 783 — CUTOVER_RUNBOOK risk item 3): NO zid is ever + blocklisted and the k-means warm start STAYS (cluster-id stability + across ticks is user-facing). Fix = (a) vectorize the warm-start + k-means hot path (per-center BLAS distance columns replacing the + per-pair python _euclidean loop; bit-identity gated by the Q11 tie + test + the full battery) and (b) a deterministic seeded sampled PCA + for extreme shapes (Clojure's large-conv graph only ever special- + cased :pca — conversation.clj:760-773 — so (a) has no Clojure + counterpart to port and (b) is the deterministic version of theirs). + 7 of 15,575 prodclone convs ever crossed the old cutoffs. 10. MOVED PRE-CUTOVER (Julien 2026-07-27: "we need to land clean code"): vectorized-code readability + blob-injection tests — PR 14b/14c from HANDOFF_PR14_VECTORIZED_REFACTOR.md (14a shipped as #2564). Now Phase 3 diff --git a/delphi/tests/test_legacy_kmeans.py b/delphi/tests/test_legacy_kmeans.py index 51331d0cf..19f9b7e4b 100644 --- a/delphi/tests/test_legacy_kmeans.py +++ b/delphi/tests/test_legacy_kmeans.py @@ -363,6 +363,21 @@ def test_euclidean_uses_clojure_cancellation_formula(self): # Normal-scale distances stay correct. assert _euclidean(np.array([0.0, 0.0]), np.array([3.0, 4.0])) == pytest.approx(5.0) + def test_euclidean_propagates_nan_instead_of_clamping(self): + """#2663 review pin: NaN input must PROPAGATE (real vectorz has no + clamp) — the former ``max(0.0, d2)`` silently returned 0.0 because + python's two-arg max returns its FIRST argument when the second is + NaN. A NaN center reaching cluster_step would otherwise be silently + absorbed as distance-0 instead of surfacing the corruption.""" + import math + + from polismath.pca_kmeans_rep.legacy_kmeans import _euclidean + + assert math.isnan(_euclidean(np.array([np.nan, 0.0]), + np.array([1.0, 2.0]))) + assert math.isnan(_euclidean(np.array([1.0, 2.0]), + np.array([0.0, np.nan]))) + def test_near_coincident_singletons_merge_to_later_cluster(self): from polismath.pca_kmeans_rep.legacy_kmeans import _NamedData, kmeans