diff --git a/delphi/docs/CLOJURE_QUIRKS.md b/delphi/docs/CLOJURE_QUIRKS.md index ba565e56b..f891a178c 100644 --- a/delphi/docs/CLOJURE_QUIRKS.md +++ b/delphi/docs/CLOJURE_QUIRKS.md @@ -29,7 +29,7 @@ Status legend: **REPLICATED** (Python legacy mode reproduces it) · | Q8 | **Meta-tid `0`-is-truthy routing bug (#1961)** — every comment took the meta branch of `priority-metric`, flattening priorities to ~all-49. | math/.../repness or priorities path; see MATH_ALGORITHM_HISTORY.md | FIXED-UPSTREAM (#2611, merged 2026-07-18; not yet in a prod deploy as of 2026-07-21) | Done; watch prod deploy | | Q9 | **Subgroup smoother was UNCLAMPED in prod** until the #2575 clamp landed via #2609 (merged 2026-07-18). Only relevant to blobs generated by pre-#2609 workers. | conversation.clj:534-567 pre-#2609 | FIXED-UPSTREAM (and moot for us given Q7 carve-out) | — | | Q10 | **Large-conv mini-batch PCA is UNSEEDED-RANDOM** — `conv-update` dispatches to `large-conv-update` when n-ptpts > 10000 OR n-cmts > 5000 (conversation.clj:784-815); its `:pca` runs `partial-pca` on a fresh Mersenne-Twister row sample PER ITERATION with NO seed (`(sampling/sample (range n-ptpts) :generator :twister)`, conversation.clj:763-773) — so two Clojure runs of the same large conversation produce different PCA (and everything downstream). There is NO deterministic reference on this path, not even Clojure itself. Verified empirically 2026-07-22: pakistan/engage/bg2050 battery entries diverge at EXACTLY the first step crossing the cutoff (n-cmts 4914→6028, 4850→5765, 4694→6701) and match on every step before it. | conversation.clj:757-773 (sample-size + large graph), 784-815 (dispatch) | CARVED-OUT of R1 certification: the battery certifies the deterministic full-PCA path by pinning `{:ptpt-cutoff :cmt-cutoff}` HUGE in the replay drivers (conv-update accepts opts; math/dev/ change only — exclusion logged per run, never silent). Python keeps full PCA at all sizes. | Poller phase must decide the production-cutover story for large convs (python full-PCA is deterministic and strictly better; document as an intentional improvement over Clojure's randomized mini-batch). | -| Q11 | **kmeans distances suffer dot-formula cancellation — near-coincident points TIE at exactly 0.0 and merge.** vectorz's `matrix/distance` on the row types kmeans actually passes (ArraySubVector view vs Vector center) computes d^2 = |a|^2+|b|^2-2ab, whose cancellation floors any true distance below ~1e-8 to EXACTLY 0.0 (verified in-process: pid-5/7 pair at true distance 4.66e-15 -> both cluster distances 0.0, while the same call with a copied row returns the true values). Ties then resolve via min-key LAST-wins -> near-coincident points collapse into the LATER cluster and the emptied cluster is dropped (vw every-vote step 57: cluster 8 absorbs [5 7], cluster 6 dropped). Python's norm(a-b) has no cancellation -> no tie -> no merge. | clusters.clj:44-52 (add-to-closest via matrix/distance); verified 2026-07-22 via math/dev/proj_probe.clj | TO REPLICATE (legacy mode): compute kmeans distances as sqrt(max(0, |a|^2+|b|^2-2ab)) in float64 — reproduces the 0.0 tie deterministically (numpy check confirms) | CORRECTED 2026-07-22 s4: `most-distal` does NOT use the cancellation formula — its rows come from `get-row-by-name` (matrix/get-row), a type on which `matrix/distance` returns the TRUE value (pc-revote-01 probe: clj's near-tie gaps are 2.5e-16/5.6e-17 = true-formula-sized, not the 1e-14 the cancellation formula produces on the same points). Python's port applies the cancellation formula at the shared helper, so py most_distal differs from clj in the low bits — benign wherever gaps exceed ~1e-8·scale; the only regime where it matters is the Q13 knife-edge class, which is carved out. Left as-is deliberately. NaN addendum (s7, #2663 review): the port lets NaN PROPAGATE through _euclidean (real vectorz has no clamp); only negative cancellation residue is floored to 0.0 — pinned by test_euclidean_propagates_nan_instead_of_clamping. | +| Q11 | **kmeans distances suffer dot-formula cancellation — near-coincident points TIE at exactly 0.0 and merge.** vectorz's `matrix/distance` on the row types kmeans actually passes (ArraySubVector view vs Vector center) computes d^2 = |a|^2+|b|^2-2ab, whose cancellation floors any true distance below ~1e-8 to EXACTLY 0.0 (verified in-process: pid-5/7 pair at true distance 4.66e-15 -> both cluster distances 0.0, while the same call with a copied row returns the true values). Ties then resolve via min-key LAST-wins -> near-coincident points collapse into the LATER cluster and the emptied cluster is dropped (vw every-vote step 57: cluster 8 absorbs [5 7], cluster 6 dropped). Python's norm(a-b) has no cancellation -> no tie -> no merge. | clusters.clj:44-52 (add-to-closest via matrix/distance); verified 2026-07-22 via math/dev/proj_probe.clj | TO REPLICATE (legacy mode): compute kmeans distances as sqrt(max(0, |a|^2+|b|^2-2ab)) in float64 — reproduces the 0.0 tie deterministically (numpy check confirms) | CORRECTED 2026-07-22 s4: `most-distal` does NOT use the cancellation formula — its rows come from `get-row-by-name` (matrix/get-row), a type on which `matrix/distance` returns the TRUE value (pc-revote-01 probe: clj's near-tie gaps are 2.5e-16/5.6e-17 = true-formula-sized, not the 1e-14 the cancellation formula produces on the same points). Python's port applies the cancellation formula at the shared helper, so py most_distal differs from clj in the low bits — benign wherever gaps exceed ~1e-8·scale; the only regime where it matters is the Q13 knife-edge class, which is carved out. Left as-is deliberately. NaN addendum (s7, #2663 review): the port lets NaN PROPAGATE through _euclidean (real vectorz has no clamp); only negative cancellation residue is floored to 0.0 — pinned by test_euclidean_propagates_nan_instead_of_clamping. Vectorization addendum (s7, PR #2679): the batched-matmul columns preserve the cancellation ties BIT-EXACTLY (dgemv/einsum were rejected for last-ulp reassociation drift; exact-== pins travel with CI). | | Q12 | **Cold-tick PCA start vector is UNSEEDED-RANDOM** (`rand-starting-vec`, pca.clj:79-82 — the original author's own 'should really throw a [seeded] random number generator in the equation here... XXX' comment). With a small eigengap the fixed 100 power iterations do NOT fully converge, so the start-dependent residual (~1e-4 on pc-smallmix-01 step 0) survives into comps/projections — even two Clojure runs differ on the cold tick. Warm ticks are unaffected (start-vectors = previous comps). | pca.clj:79-101 | CARVED OUT for certification: BOTH replay drivers pin the cold start to the ONES vector — the exact value power-iteration already pads new-comment columns with (pca.clj:46-49 / pca.py _power_iteration) — via a single-element [1.0] start that padding expands to any width (dev/replay.clj certify-cold-start-pca + replay/driver.py seed). Logged per run. | Production: seed the start vector (the author's own XXX) — deterministic cold ticks with no behavior change at convergence | | Q13 | **Warm-chain split-loop extraction order is knife-edge-chaotic on tie-dense geometry.** `clean-start-clusters`' split loop (clusters.clj:250-273) extracts the most-distal point one at a time; when several near-coincident candidates tie (within-engine distance gaps at the few-ulp level, e.g. 2.5e-16/5.6e-17 on pc-revote-01 step 1 among 4 pids), the extraction ORDER — and thus the minted singleton ids and final partition — is decided by sub-ulp arithmetic noise. Cross-engine, projections differ at ~1e-5 (residual small-eigengap power-iteration noise, tolerant-accepted), ELEVEN orders above the gaps: no arithmetic replication can reproduce Clojure's order (verified 2026-07-22: both probes extract the identical 28-pid sequence, then clj picks {82,99} where py picks {80,99} from the tied set {80,82,99,108}; even the true-distance formula ranks them differently per engine). Same irreducibility class as the vw every-vote step-57 knife edge. | clusters.clj:202-217 (most-distal), 250-273 (split loop); probes: math/dev/proj_probe.clj split-probe + delphi/scratch/probe_revote_split.py | CARVED OUT: battery keeps revote coverage via pc-revote-02 (34 ptpts, ~28% revotes — rich geometry, no knife edge; MATCH 6/6 first try) in place of pc-revote-01 (205 ptpts on 15 comments — near-discrete projection space). Ledgered on FP-912391ece7/FP-c29173e1ba/FP-98dc728043. | Not a Clojure bug — an irreducible float-chaos regime. Any future dataset that diverges ONLY in split-loop extraction order on few-ulp gaps belongs to this class: probe with split-probe, then swap or document. | | Q14 | **No small-dimension guards — tiny matrices run the real math.** Clojure runs powerit PCA on 1x1/1xN/Nx1 rating matrices (real center, comps rank-capped at min(rows,cols)) and `conv-repness` always returns its best-agree comment even for degenerate shapes — where pre-parity Python short-circuited: PCA returned zeros (center -0.0 vs clj -1.0 on a 1x1 matrix; comps zero-padded to 2 vs rank-capped 1) and `conv_repness` returned empty repness/consensus below shape 2. | pca.clj (powerit-pca, no dim guard); repness.clj conv-repness (best-agree fallback) | REPLICATED (2026-07-22, PR #2653: legacy mode relaxes the `<2` guards in conversation.py::_compute_pca, pca.py::pca_project_dataframe, repness.py::conv_repness; single-vote fixtures with exact Clojure-derived values, tests/test_legacy_blob_shape.py; certified end-to-end by the vw-every-vote-56 battery entry whose early steps are 1xN). Ledgered late — flagged by the 2026-07-22 s4 review pass; row added then. | `improved` mode keeps the guards (sane early returns) | diff --git a/delphi/docs/CUTOVER_RUNBOOK.md b/delphi/docs/CUTOVER_RUNBOOK.md index d73262cc3..467495817 100644 --- a/delphi/docs/CUTOVER_RUNBOOK.md +++ b/delphi/docs/CUTOVER_RUNBOOK.md @@ -1,19 +1,26 @@ # Clojure→Python math cutover runbook -Written 2026-07-26 (post R1-parity DONE, GOAL_STATE.md). Companion: -MATH_POLLER_DESIGN.md §4 (phases), MATH_POLLER_EQUIV_SPEC.md (the live -equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19). - -## Evidence base (what is PROVEN as of 2026-07-26) - -- R1 battery: 20/20 MATCH on four consecutive clean pass-pairs (2026-07-24), - ledger zero open. Warm chains, moderation, meta, bans, revotes, degenerate - ticks, restart seams. -- Live poller equivalence vs the REAL Clojure container: vw 8/8 batches + - pc-meta-02 6/6 batches MATCH (moderation stream live, kill+restart seam, - tick/watermark semantics, bidToPid/ptptstats row-identical; floats within - the measured clj self-jitter envelope). -- Full delphi suite green (1134 passed at goal close; CI green at stack tip). +Written 2026-07-26 (post R1-parity DONE); refreshed 2026-07-28 (s7, post +GOAL_CUTOVER_READY DONE). Companion: MATH_POLLER_DESIGN.md §4 (phases), +MATH_POLLER_EQUIV_SPEC.md (the live equivalence protocol), +CLOJURE_QUIRKS.md (Q1-Q19), POST_CUTOVER_IMPROVEMENTS.md (the queue). + +## Evidence base (what is PROVEN as of 2026-07-28, s7) + +- MODE COLLAPSE LANDED (#2665-#2671): the engine has ONE code path — + exact legacy semantics; flag machinery deleted; improved branches + parked (improvements/* bookmarks); bans deleted (not a Polis feature). +- Battery: 20/20 MATCH pairs re-certified at every s7 milestone — + post-collapse, post-refactor (#2673), post-vectorization (#2679), and + on the final tree; divergences ledger 81 entries, 0 open. +- Live poller equivalence vs the REAL Clojure container, RE-RUN on the + collapsed tree: vw 8/8 + pc-meta-02 8/8 batches MATCH, non-vacuous, + 0 envelope-excused divergences (moderation stream, kill+restart seam, + bidToPid/ptptstats row-identical). +- Goldens re-recorded at the collapse tree (verify-then-record; comparer + 7/7 PASS); full delphi suite green (1171 passed at s7 close). +- Clarity refactor 14b/14c (#2673) + vectorized warm-start kmeans + (#2679, bit-identical, warm tick ~70x) landed pre-cutover. ## What is NOT yet proven (risk register) @@ -26,16 +33,21 @@ equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19). (documented improvement, same blob shape, server-compatible). A shadow comparer WILL flag these convs — expected, not a defect. Decide the acceptance for them up front (structural-only, or exclude from compare). -3. **Poller throughput**: ~1.66 ticks/s per process — measured on EC2 - (r8g.4xlarge, cost-model study) on biodiversity-sized replays. - **PRE-FLIP MEASUREMENT DONE (2026-07-27, s7)** — one full-PCA tick of - the largest prodclone shape (33,422 ptpts x 783 cmts, 2.0M votes; - synthesized, seeded) on r8g.4xlarge via - scripts/large_conv_tick_bench.py: - **cold tick 519.6s (~8.7 min); WARM (steady-state) tick 1856.0s - (~30.9 min)** — the warm tick is ~3.6x the cold one (legacy kmeans - lineage warm-start dominates). Local M-series cross-check: 430.8s / - 2095.3s — same order, so this is algorithmic, not instance-bound. +3. **Poller throughput**: ~1.66 ticks/s per process on biodiversity-sized + replays — measured on EC2 (r8g.4xlarge, cost-model study) with the + PRE-VECTORIZATION engine, so it is now a stale LOWER BOUND (#2679 + speeds up every conv's k-means, not just giants; re-measure during + the shadow soak if a capacity number is needed). + **PRE-FLIP MEASUREMENT (2026-07-27 s7, pre-vectorization — + SUPERSEDED by the verdict below)** — one full-PCA tick of the largest + prodclone shape (33,422 ptpts x 783 cmts, 2.0M votes; synthesized, + seeded) on r8g.4xlarge via scripts/large_conv_tick_bench.py: + cold tick 519.6s (~8.7 min); WARM (steady-state) tick 1856.0s + (~30.9 min). The then-observed "warm ~3.6x cold" asymmetry was an + artifact of the un-vectorized port's per-pair python loops in the + lineage warm start — ELIMINATED by #2679 (post-vectorization the two + are within ~10%: 29.0s vs 26.6s). Local M-series cross-check ran + same-order both times (430.8s/2095.3s before; 28.2s/26.7s after). **VERDICT (FINAL, 2026-07-28 s7): serial is OK at every observed shape.** Item 9a (vectorized warm-start k-means, PR #2679 — bit-identical: exact-== pins vs the scalar reference, knife-edge Q11 @@ -66,14 +78,57 @@ equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19). python's per-zid FIFO+lock design does not have it (equivalence runs verified py carries the full vote stream). +## Execution shape (s7 rulings + analysis — read before Step 0) + +**Shadow vs clean replace (analysis 2026-07-28; decision pending +Julien):** recommended = TIME-BOXED SHADOW, 24-48h, exit checklist +below. Rationale: it tests the only untested dimension (real prod +churn/concurrency/dirty data) at near-zero complexity — the service, +env var, and compare machinery all exist; the time box kills +shadow-limbo risk. Clean replace is defensible on the evidence +(bit-exact battery + live equivalence) and rollback stays cheap +(restart `math`; caching_tick is MAX+1 both ways), but forfeits the +baseline rows that make subtle math weirdness detectable. Memory is a +non-issue either way: host 128 GiB, python capped 16g (set +MATH_CONV_CACHE_CAP — and keep it set in ANY long-running deployment, +not just the soak; eviction cost = the certified restart seam). +Shadow exit checklist (agree BEFORE starting): rows advancing on all +active zids; zero parked zids / errorconv dumps; spot-compare N active +zids structurally identical (poller_equiv comparer on row pairs); +large-conv divergence dismissed per risk 2/Q10. + +**One WIP PR per step (for a future session):** +- PR-S0 (promote): get the stack onto `stable` (prod deploys track + stable, not edge — after_install.sh pulls stable). +- PR-S1 (shadow): scripts/after_install.sh math role line → + `up -d math math-python`; add MATH_CONV_CACHE_CAP + MATH_PYTHON_ENV + to the SSM-sourced .env (polis-web-app-env-vars secret); exit + checklist copied into the PR body. +- PR-S2 (flip): ONE mechanism (ruling needed: poller MATH_ENV→'prod' vs + server mathEnv→'python'); revert instructions in the PR body. +- PR-S3 (decommission): remove `math` from compose + its + after_install.sh line; archive note for the Clojure tree. + +**CDK impact: NONE required for steps 0-3.** The python poller runs on +the existing math-worker host (r8g.4xlarge, MathWorkerLaunchTemplate) +via compose; same Postgres path/security groups; math_writer needs no +new IAM (Postgres only); CodeDeploy math deployment group unchanged +(the only deploy-side edit is after_install.sh, which ships with the +repo). Verified: cdk/ec2.ts, cdk/launchTemplates.ts, appspec.yml, +scripts/after_install.sh. CDK would only enter later if the math host +itself is retired/resized post-decommission (candidate: downsize +r8g.4xlarge once the vectorized engine's real utilization is known — +measure during the soak first). + ## Step 0 — land the stack (morning) -1. Triage the 18 Copilot reviews requested overnight (2026-07-26) on - #2641-#2658; apply/reply per the standing triage rules. -2. Confirm CI green: stack-tip python-ci dispatch + PR checks (see - spr status; #2648's mid-stack red is a stack-position artifact — the - same tests pass from #2656 upward — cosmetic for deploy, which builds - the tip). +1. Reviews: DONE through #2682 (Copilot triage s6 = #2663; Copilot + credits exhausted since — all later PRs reviewed by independent + review agents, all sound; findings applied). Nothing outstanding. +2. Confirm CI green at the stack tip (python-ci workflow_dispatch on the + tip branch; every s7 dispatch was green). Historic note: a mid-stack + red (e.g. #2648-era) is a stack-position artifact — deploy builds the + tip. 3. Merge bottom-up: `jj spr merge --count ` (spr handles squash order). NEVER the GitHub UI. Then a normal edge deploy. @@ -115,7 +170,7 @@ small/mid convs; large-conv divergence understood per risk #2. One env change, instantly reversible: - Set the python poller's MATH_ENV to the server's Config.mathEnv ('prod'); stop the clojure `math` service. (Or flip the server's MATH_ENV to - 'delphi' — pick ONE mechanism and write it down.) + 'python' — pick ONE mechanism and write it down.) - Watch: TS prefetch (pca.ts caching_tick > last, ~2.5s poll) keeps serving; nextComment routing gets comment-priorities; participants bidToPid present. @@ -125,7 +180,9 @@ coexist; nothing is destroyed by the flip in either direction. ## Step 3 — decommission (later) -Remove the `math` service from compose/deploy; archive the Clojure tree -(it remains the R1 oracle). Follow-ups parked in the journal: equiv-in-CI -decision, improved-mode ban coverage, fraction-cut py-round fix, quirk -un-replication in improved mode (the post-cutover engine option). +Remove the `math` service from compose/deploy (and its `up -d math` +line in scripts/after_install.sh); archive the Clojure tree (it remains +the certification oracle). Follow-ups now live in +POST_CUTOVER_IMPROVEMENTS.md (items 2-9b, 11, 12 — quirk un-replication, +warm-start persistence, optional seeded sampled PCA) plus the journal's +equiv-in-CI decision and the fraction-cut py-round fix. diff --git a/delphi/docs/MATH_POLLER_DESIGN.md b/delphi/docs/MATH_POLLER_DESIGN.md index ca055e8bb..20c3dae80 100644 --- a/delphi/docs/MATH_POLLER_DESIGN.md +++ b/delphi/docs/MATH_POLLER_DESIGN.md @@ -82,7 +82,7 @@ math_env string written), `VOTE_POLLING_INTERVAL` (ms, default 1000), 1. **Shadow (this PR):** new compose service `math-python` (profile-gated, `--profile math-python`; renamed s7 from delphi-math-poller) running next to the Clojure `math` service, writing under a - DIFFERENT `math_env` (e.g. `MATH_ENV=delphi` while Clojure writes `prod`/`dev`). + DIFFERENT `math_env` (e.g. `MATH_ENV=python` while Clojure writes `prod`/`dev`). `UNIQUE(zid, math_env)` makes the rows invisible to the prod server. No consumer change, zero production risk. 2. **Parity monitoring:** a comparer job diffs Python-vs-Clojure `math_main` rows per diff --git a/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md b/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md index b121d0c5f..fb86c2a30 100644 --- a/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md +++ b/delphi/docs/POST_CUTOVER_IMPROVEMENTS.md @@ -52,7 +52,11 @@ snapshots at the collapse commit. Each PR: change + tests + re-certified outputs + a CHANGELOG-quality description. Sources: CLOJURE_QUIRKS "Later fix" column, journal parked -items. +items. The mode-collapse deletions for items 2/4/5/8 are parked VERBATIM +as jj bookmarks improvements/item-{2-degenerate-guards,4-mod-watermark, +5-current-group-votes,8-modern-solvers} (pushed to origin; reverse +patches of the collapse commits — re-landing = keep the improved side, +the flag refs inside are dead by design). 1. ~~Honor participant bans~~ — DROPPED ENTIRELY (Julien 2026-07-27): bans are not a Polis feature (201 rows ever, never honored by any @@ -79,18 +83,23 @@ items. by Julien s7 after the EC2 measurement: r8g.4xlarge warm tick 1856s at 33,422 x 783 — CUTOVER_RUNBOOK risk item 3): NO zid is ever blocklisted and the k-means warm start STAYS (cluster-id stability - across ticks is user-facing). Fix = (a) vectorize the warm-start - k-means hot path (per-center BLAS distance columns replacing the - per-pair python _euclidean loop; bit-identity gated by the Q11 tie - test + the full battery) and (b) a deterministic seeded sampled PCA - for extreme shapes (Clojure's large-conv graph only ever special- - cased :pca — conversation.clj:760-773 — so (a) has no Clojure - counterpart to port and (b) is the deterministic version of theirs). + across ticks is user-facing). + (a) vectorize the warm-start k-means hot path — **DONE PRE-CUTOVER + (s7/2026-07-28, PR #2679)**: batched-matmul BLAS columns, BIT-IDENTICAL + (exact-== pins vs the scalar reference incl. the Q11 knife-edge tie; + battery 20/20 x2); re-measured on the same r8g.4xlarge: warm tick + 1856.0s -> 26.6s (~70x), cold 519.6s -> 29.0s. Final verdict in the + runbook: serial OK at every observed shape. + (b) deterministic seeded sampled PCA for extreme shapes — now + OPTIONAL (hygiene/further speedup, no throughput need; Clojure's + large-conv graph only ever special-cased :pca — + conversation.clj:760-773 — so (b) is the deterministic version of + theirs and (a) had no Clojure counterpart to port). 7 of 15,575 prodclone convs ever crossed the old cutoffs. -10. MOVED PRE-CUTOVER (Julien 2026-07-27: "we need to land clean code"): - vectorized-code readability + blob-injection tests — PR 14b/14c from - HANDOFF_PR14_VECTORIZED_REFACTOR.md (14a shipped as #2564). Now Phase 3 - of GOAL_CUTOVER_READY.md, battery-guarded; listed here for lineage only. +10. MOVED PRE-CUTOVER and **DONE** (s7, PR #2673): vectorized-code + readability (14c two-phase split) + blob-injection tests (14b) from + HANDOFF_PR14_VECTORIZED_REFACTOR.md (14a shipped as #2564); + battery-proven bit-identical. Listed here for lineage only. 11. Cleanup pass: the deferred cosmetic simplifications (e.g. the #2644 legacy reindex no-op), dead update_moderation seams, the Q7 subgroup computation deletion upstream if the Clojure tree is still around. @@ -101,3 +110,18 @@ items. nondeterminism) — resolved by replacement, not repair. - Q19 (Clojure conv-actor race): moot at decommission; the Python design (per-zid FIFO + lock) is the fix. + +12. **Persist the full warm-start state across restarts** (s7, from the + conv-cache discussion): the math_main whitelist (conv_man.clj:52-74 + parity) omits the per-k group_clusterings map and the group-K + smoother state, so ANY restore (restart, LRU eviction, redeploy) is + a partial warm start: the next tick's per-k group k-means runs COLD + (its warm-start input is empty for one tick) and the smoother resets + — meaning it immediately re-accepts the current best k instead of + damping, so a restart can flip K where a continuous run would not. + This is CERTIFIED Clojure-faithful restart behavior (battery restart + entries MATCH; Clojure lost the same state on its 4-hourly reboots), + NOT a numbered quirk today. Improvement: persist group_clusterings + + smoother state (blob keys or sidecar) so restores are fully warm and + restart-induced K flips disappear. Blob-schema addition — verify + server tolerance for extra math_main keys before landing. diff --git a/delphi/polismath/poller/__init__.py b/delphi/polismath/poller/__init__.py index b293ebf5d..77480263b 100644 --- a/delphi/polismath/poller/__init__.py +++ b/delphi/polismath/poller/__init__.py @@ -158,7 +158,7 @@ # Single poll cycle then exit (smoke test / cron-style) uv run python scripts/math_poller.py --once -Shadow-mode deployment writes under a DISTINCT ``MATH_ENV`` (e.g. ``delphi``) +Shadow-mode deployment writes under a DISTINCT ``MATH_ENV`` (e.g. ``python``) next to the Clojure ``math`` container; ``UNIQUE(zid, math_env)`` keeps the rows invisible to the prod server until cutover. """ diff --git a/docker-compose.yml b/docker-compose.yml index a4dbfa8a9..b3263411c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -149,7 +149,7 @@ services: # container. Profile-gated so it only runs when explicitly requested # (`--profile math-python`). Reuses the delphi build target and overrides the # command to run the poller CLI. Defaults to SHADOW mode: writes under a - # DISTINCT math_env (`delphi`) so its rows are invisible to the prod server + # DISTINCT math_env (`python`) so its rows are invisible to the prod server # (UNIQUE(zid, math_env)) — zero production risk while parity is validated. # See delphi/docs/MATH_POLLER_DESIGN.md §4. math-python: