diff --git a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md index e25950760..de0a3cb35 100644 --- a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md +++ b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md @@ -2002,3 +2002,209 @@ stability fix (`jc/clj-parity-d1-pca-sign-flip-prevention`) — that is temporal **What's next:** with D1b closed, the remaining blockers on the D12 un-mirror are the D1 sign-stability work and the participant-filtering / vote-copy reconciliation noted above. + +## Session: Overnight orchestration — replay harness H-A, sequential-bits A/B/D′, input-fidelity fixes (2026-07-17→18) + +Host session "Fable-Pyclj-Parity". Julien handed over for the night with a new +directive: proceed autonomously on math-core with careful per-change notes and a +morning walkthrough (recorded in project memory; see +`scratch/MORNING_WALKTHROUGH_2026-07-18.md` for the full walkthrough). All PRs +opened as **Drafts** per mid-session instruction. Work executed by opus/sonnet +subagents in isolated git clones, integrated serially by the session integrator +with a full-suite gate per commit. + +### RETRACTION: "in-conv 67 vs 68-69" (2026-07-17 entry above) was a journal error + +Live rerun on the full 4683-row vw CSV: Python cold-start in-conv = **67**, +matching the Clojure cold-start blob **pid-for-pid** (excluded: pid 13 with 5 +votes, pid 37 with 3 — both below min(7, 125)). The "68" was Clojure's +*incremental* blob (monotonic in-conv admitted pid 13 when n_cmts, hence the +threshold, was still small — the already-xfailed incremental-vs-cold distinction +from PR #2421); "69" is the raw voter count. The D12 un-mirror chain therefore +has NO participant-filtering blocker; what remains is the generator vote-copy +delta (fixed this session, below) and extremity/PCA parity. + +### Input fidelity: cold-start generator now copies FULL revote history + +`generate_cold_start_clojure.py::copy_votes_with_fresh_timestamps` used +`DISTINCT ON (pid, tid) ... ORDER BY created DESC`, silently dropping superseded +revote rows — on vw exactly 128 of 4683 (87 revoted pairs: 63×2 + 14×3 + 4×4 + +5×5 + 1×6 = 4683−4555). Both engines implement later-vote-wins internally, so +dedup-at-source only harmed input parity and erased revote dynamics +(REPLAY_HARNESS_DESIGN.md §5 explicitly forbids it). Now copies every row +`ORDER BY created ASC, ctid ASC` with strictly-increasing 10 ms fresh timestamps +(ctid tiebreak ≈ insertion order for same-ms revotes; the true relative order of +same-ms revotes is ambiguous in the source itself — documented in the +docstring). Timestamp-ordering audit: prepare_votes_data still loads file order +(2136 adjacent inversions in vw) but "file-order-last" vs "timestamp-last" +produces **0** value differences on vw's 87 revoted pairs — latent quirk, not a +manifesting bug; the replay slicer sorts properly (below). + +### Replay harness Phase H-A — BUILT (pure spine, no math-core changes) + +`polismath/replay/` gains `schedule.py` (spec JSON, 4 cut modes, 6 presets, +slicer with timestamp sort + input-order tiebreak, revotes kept), `driver.py` +(fold of `update_votes` batches with recompute at cuts; per-step `to_dict()` blob ++ cheap diagnostics), `store.py` (`real_data/.local/replays///` +with full provenance incl. vote-sign convention), `stepcompare.py` +(ConversationComparer repointed to step-vs-step), `scripts/replay_driver.py` +CLI (run/compare). `types.py`/`real_data.py` lifted **byte-identically** from the +R2 branch (`jc/r2-schedule-inference`) so its rebase dedups. 46 new tests in +`tests/replay_harness/`. Verified: the only nondeterministic blob field is +`math_tick` (wall clock); everything else is bit-identical across runs. +Documented deviations: moderation applied as cumulative state at vote cuts (no +mod-triggered cuts yet); tail after last cut dropped (use an `"end"` cut). +Seam wishlist for future PRs: expose per-k silhouettes; injectable math_tick; +`update_moderation` cannot clear a set with an empty list; `last_updated=0` +falls back to wall-clock; lean-blob mode for R2 search perf. + +### Sequential-bits port — spec + first three increments (math-core) + +Full inventory of Clojure's cross-tick state now in +`docs/SEQUENTIAL_BITS_PORT_SPEC.md` (verified file:line for all 11 behaviors). +Headline spec findings: (a) the **#2575 subgroup clamp is NOT in Clojure HEAD** +— it is open PR #2609, so current production Clojure runs the unclamped +subgroup smoother; (b) **new uncatalogued divergence**: Clojure's +`:comment-priorities` reads the **previous tick's** `(:group-votes conv)` +(conversation.clj:650), Python uses the current tick's — masked today by the +#2571 mirror, must be honored at un-mirror time; (c) Python computes no +subgroups at all, so subgroup-level ports are latent. + +Landed (each behind `POLISMATH_ENGINE_MODE=clojure-legacy`; default `improved` +mode verified byte-identical — hard gate): + +- **PR-A**: engine-mode flag (`polismath/utils/engine_mode.py`), cold-default + `group_clusterings`/`group_k_smoother` fields, `recompute()` prev-tick state + capture threaded as parameters (mirrors Clojure fnks reading the incoming conv). +- **PR-B**: PCA warm start — prev tick's unit comps → `powerit_pca(start_vectors=…)` + (conversation.clj:385 → pca.clj:98; 1-padding for new comments already in the + powerit port). Legacy mode requires powerit (sklearn cannot inject start + vectors; warn + fallback, never silent). +- **PR-D′**: group-k-smoother as a pure function + (`pca_kmeans_rep/group_k_smoother.py`): buffer=4 consecutive-agreement rule, + #2536 stale-k clamp, and Clojure `max-key` HIGHER-k-wins tie-break (improved + mode's lower-k-wins strict `>` untouched). Known deliberate gap: degenerate + ticks (<2 in-conv / <2 base clusters) preserve rather than reset smoother + memory; clamp protects the next real tick. + +Suite: 406 → 480 passed (46 harness + 28 seqbits tests), 17 skipped, 47 +xfailed, 0 failed at every integration step. + +### First gap measurement — D1 sign-flip captured on real data + +vw, uniform 8-cut schedule, improved vs clojure-legacy: steps 0–5 bit-identical; +at **step 6 improved (cold) mode flips PC2's sign** (66 exact mismatches, all +±y at rel_diff 200%) while the legacy warm-started chain holds orientation; by +step 7 the flip cascades into genuinely different group-cluster geometry (964 +exact + 1095 tolerant mismatches). Legacy mode is bit-for-bit deterministic +across independent runs (0/8 divergence) — the property R2's forward model +requires. **D1 conclusion:** in legacy mode, sign stability is delivered by the +warm-start chain (Clojure's own mechanism — it has no explicit alignment +either). Improved-mode cross-restart sign alignment would need persisted prev +comps — deferred with a design note. + +### Golden kit (polis-algo-research) — evaluated for lift + +The algorithms-report repo's `golden-kit` (pre-bug Clojure oracle, +`polis-math:prebug-8f278034`, 21 fixtures × 5-repeat ensembles, 5-component +certification suite) was evaluated empirically against the CURRENT main-repo +tree: 4 of 5 suites pass unchanged (`repness.py`/`clusters.py` byte-identical +to its frozen reference); the comment-extremity tests break **because the kit +still assumes the pre-D1b buggy convention** — independent confirmation that +D1b fixed a real bug (the kit's own sign-convention dossier had recommended +exactly this fix). Its 5-repeat ensembles are ready-made cold-start self-jitter +tolerance floors for §9 of the replay design. Lift decisions left to Julien +(217 MB goldens → LFS/gzip/thinning; orphaned frozen-source pin must move to +live tree; fixture naming vs discover_datasets(); missing CC-BY notice). + +### H-B — Clojure Mode A driver: DONE (same night) + +`math/dev/replay.clj` (416 lines) + `:replay` deps.edn alias; pure in-process +conv-update reduce over schedule JSON, per-step `prep-main` blob capture +(23-key EXACT match with the committed vw cold-start math blob), `--repeats` +self-jitter mode, `--edn` full-state dumps; cross-language shim so +`stepcompare` diffs clj-vs-py recordings unchanged (5 tests). `math/src/` +untouched. Key results: +- **Self-jitter floors (§9) measured**: pca.comps repeat-to-repeat ~1e-4 at the + cold step 0, collapsing to ~1e-8/1e-9 under the warm-start chain; every + non-PCA field bit-identical between repeats. +- **First true Python↔Clojure gap measurement** (3-cut vw): tid / in-conv / + base-cluster-id SETS identical at every step; base-cluster x-coords are + near-exact NEGATIVES (mean |clj+py| = 7.5e-5) — same geometry up to + reflection. Genuine numeric gap confined to PCA cells (443, widening per + step) + downstream group-aware-consensus. +- **Blob-shape deltas catalogued** (gates the poller FLIP phase): Python + to_dict emits scalar votes-base/group-votes where Clojure emits + per-base-cluster vectors; `comment_priorities` vs hyphenated + `comment-priorities`; Python-only extra keys (proj, moderation, vote_stats, + math_tick); tids/in-conv ordering. Needs a math_main-exact serializer. +- Driver gotchas: `-i dev/replay.clj` (not a classpath dir) avoids the + `dev/user.clj` :dev-deps trap; cheshire requires CoreMatrixBooter's + vectorz encoders registered before serializing conv state. + +### PR-C + PR-E — base-cluster lineage + in-conv carry: DONE (same night) + +PR-C: `pca_kmeans_rep/legacy_kmeans.py` (468 lines) — faithful numpy port of +clusters.clj k-means with id lineage (init-clusters first-k-distinct; +clean-start-clusters = safe-recenter drop-vanished + big-cluster fallback, +uniqify identical centers with merge-keeps-larger-cluster's-id (tie → later +arg, matching max-key), most-distal split with `(inc max-id)` ids; +cluster-step drop-empty; same-clustering? sorted centers < 0.01 with +zip-truncation). Wired legacy-only: base level warm-starts from prev +base_clusters (base-iters=100); group level per-k warm-started over +weighted base-cluster centers. **Port-discovered Clojure fact: the group level +actually runs max-iters=20** — `kmeans` never destructures the `:cluster-iters` +key it is passed (clusters.clj:303), so Clojure silently uses the default; +mirrored as GROUP_LEGACY_ITERS=20. Legacy-mode `group_clusterings` stores +id-carrying dicts (improved keeps its tuple flow, untouched). +Cold-start invariance measured on vw: structurally bit-identical to improved +at both levels; center coords differ only ~1e-13 (np.average vs sklearn +centroid arithmetic). On degenerate near-duplicate projections legacy keeps +exact-init singletons where sklearn Lloyd collapses a pair — legacy is the +Clojure-faithful side. + +PR-E: legacy-only persistent `in_conv` carry + the greedy top-15 floor +(conversation.clj:243-269), both previously missing. **Clojure's greedy +tie-break is genuinely non-deterministic** (`sort-by` over a hash-map); +mirrored with a deterministic stable sort keyed on matrix row order — +flagged as a surrogate decision for review. + +Replay smoke: legacy mode achieves **100% base-cluster id stability** across +vw cuts vs 95.3% (dipping to 85%) improved — the lineage effect, measured. + +Suite after full integration: **525 passed / 17 skipped / 47 xfailed** +(= 406 start-of-night baseline + 119 new tests, 0 regressions all night). + +### Python math poller phase 1: DONE (same night) + +Per `MATH_POLLER_DESIGN.md` (recon-verified: production Clojure container = +poller-system ONLY; exports/report-tasks dormant or server-covered). Shipped: +`polismath/poller/` (watermark loops mirroring poller.clj:12-37; per-zid +FIFO+single-owner serialization with take-all!/split-batches coalescing; +math_writer with ONE math_tick shared across math_main / math_bidtopid / +math_ptptstats and the Clojure-exact `caching_tick = COALESCE(MAX+1,1)` +upsert the TS prefetch poll depends on), `scripts/math_poller.py` CLI, +`delphi-math-poller` compose service (profile-gated, shadow MATH_ENV). +Postgres-layer fixes en route: the dead-code writers routed through a +COMMITTING `engine.begin()` path (the old `engine.connect()` silently rolled +back INSERTs — caught by the integration test, not the mocks), atomic +math_ticks upsert, global `poll_votes_since`/`poll_moderation_since`, and +`poll_votes` now ORDERs BY zid,tid,pid,created (Clojure conv-poll parity — +row order seeds base-cluster ids). 44 unit tests + 1 opt-in integration test +that RAN against a throwaway postgres:17 (:5435): end-to-end +poll→compute→write, shadow math_env isolation, shared tick, caching_tick=1 +first write, restart-resumes. load-or-init finding: `from_dict` restores +pca/proj/moderation/stats but NOT matrices/base_clusters → full-history +rebuild on first touch (Clojure-restart-equivalent), PCA warm-seeded +opportunistically. Cutover: SHADOW ONLY until blob-shape alignment (see H-B +deltas) closes; flip = one MATH_ENV change; then Clojure decommission. + +**End-of-night suite: 570 passed / 17 skipped / 47 xfailed** — from the +406 start-of-night baseline: +164 new tests, 0 regressions, 0 xfail changes. + +### What's Next +2. R1 certification runs (Python-legacy vs Clojure CCRs) once H-B lands. +3. D12 un-mirror chain, now unblocked pending: blob regen with the fixed + generator + extremity verification; prev-tick group-votes port (spec row 7). +4. #2609 (subgroup clamp) merge decision — Clojure side, Julien's call. +5. Golden-kit lift decision — Julien's call. diff --git a/delphi/docs/MATH_POLLER_DESIGN.md b/delphi/docs/MATH_POLLER_DESIGN.md new file mode 100644 index 000000000..b7f45ddc4 --- /dev/null +++ b/delphi/docs/MATH_POLLER_DESIGN.md @@ -0,0 +1,116 @@ +# Python Math Poller — Clojure math-container replacement design + +**Status:** Design + phase-1 implementation — 2026-07-18 (overnight session "Fable-Pyclj-Parity") +**Recon basis:** every file:line below verified against the working tree on 2026-07-18 +(full recon in journal session entry). Companions: `SEQUENTIAL_BITS_PORT_SPEC.md` (the +warm-started engine this poller hosts), `REPLAY_HARNESS_DESIGN.md` (validation), +`STORAGE_V2_DESIGN.md` (provenance plumbing, to be wired when its stack lands). + +## 1. Goal + +A Python service that **completely replaces the Clojure math Docker container and its +poller**: polls Postgres for votes/moderation, maintains per-conversation math state +in-memory with the warm-started sequential engine, and writes the same four Postgres +tables the TS server and legacy clients consume — first in shadow mode next to Clojure, +then as the only math worker. + +## 2. What the Clojure container actually does (verified) + +Production runs `clojure -M:run full` (`math/bin/run:11`) = **poller-system only** +(`system.clj:51-58` — darwin/task components commented out): config, logger, +core-matrix-boot, postgres pool, conversation-manager, **vote-poller, mod-poller**. + +Responsibilities checklist: + +| Duty | Clojure | Disposition | +|---|---|---| +| Vote poll: `SELECT * FROM votes WHERE created > watermark ORDER BY zid,tid,pid,created`, ~1s cadence, group by zid, advance watermark to max(created) | poller.clj:12-37, postgres.clj:132-145 | **replace** | +| Mod poll: `SELECT * FROM comments WHERE modified > watermark`, same loop | postgres.clj:148-161 | **replace** | +| Per-zid serialized actor: coalesce queued batches (`take-all!`), process `[:votes :moderation]` in order, retry-chan (buf 10), errorconv EDN dump on failure | conv_man.clj:291-388 | **replace** | +| `math_main` upsert with **`caching_tick = (SELECT max(caching_tick)+1 … WHERE math_env=?)`** | postgres.clj:323-338 | **replace — fidelity-critical** (TS prefetch polls `caching_tick > last` every ~2.5s, pca.ts:84-151) | +| `math_bidtopid` upsert (bid→pids map, prep-bidToPid conv_man.clj:35-40) | postgres.clj:369-380 | **replace — net-new writer** (server bidToPid/bid/getPidsForGid depend on it) | +| `math_ptptstats` upsert | postgres.clj:350-361 | replace | +| `math_ticks` atomic increment (`ON CONFLICT … math_tick+1 RETURNING`) | postgres.clj:292-295 | **replace — must be atomic**, not read-modify-write | +| `math_profile` telemetry | postgres.clj:340-348 | scope out | +| Boot: watermark starts `POLL_FROM_DAYS_AGO=10` back; conv `load-or-init` from `math_main` + full vote rebuild; 4h JVM reboot (`timeout -s KILL 14400`) | poller.clj:15, conv_man.clj:188-207, bin/run | replicate load-or-init + configurable window; **no 4h reboot** (JVM workaround, not semantics) | +| CSV export (darwin→S3), report correlation, `update_math` task | tasks.clj, export.clj | **scope out**: not started in `full`; CSV export fully served by `server/src/routes/export.ts` + `report.ts`; correlation is a no-op ("No longer supported", conv_man.clj:209-219) | + +Consumers pinned: `pca.ts:98` (caching_tick prefetch), `pca.ts:360` (`WHERE zid=? AND +math_env=?` — `Config.mathEnv` selects rows), `nextComment.ts:70-119` +(comment-priorities routing), `participants.ts:6-23` (bidToPid), `report.ts` (server CSV +export reads getPca!), client-participation + client-report via `/api/v3/math/pca2`. + +## 3. Architecture (phase 1) + +``` +scripts/math_poller.py (CLI) + └─ polismath/poller/service.py MathPollerService + ├─ watermark loops (threads): votes (created>wm), moderation (modified>wm) + │ reuse PostgresClient.poll_votes / poll_moderation (already flip signs + │ to Delphi convention at ingress) + ├─ per-zid dispatch: ConversationWorkerPool + │ one FIFO queue + lock per zid → strict serialization per conversation, + │ coalescing (drain queue, merge batches, votes-then-moderation order, + │ mirroring take-all!/split-batches), bounded pool across zids + ├─ engine: Conversation chain in-memory (update_votes/update_moderation → + │ recompute) — POLISMATH_ENGINE_MODE honored; recommend clojure-legacy + │ during shadow/parity phases (warm-started PCA + k-smoother = the + │ sequential semantics Clojure has; improved mode is the post-cutover + │ option, pending persisted-comps sign stability) + ├─ load-or-init: on first message for a zid, restore from math_main + │ (from_dict) + rebuild rating matrices from full vote history + │ (conv-poll offset 0 analog), mirroring conv_man.clj:188-207; + │ non-persisted warm state (smoother counters) cold-starts, exactly + │ like a Clojure worker restart + ├─ writer: polismath/poller/math_writer.py + │ math_main (Clojure-exact caching_tick=MAX+1 SQL), math_bidtopid + │ (derived from base_clusters like prep-bidToPid), math_ptptstats, + │ atomic math_ticks; all under one math_env string; one tick value + │ shared across the writes (Clojure writes all three with the same + │ math_tick, conv_man.clj:158-169) + └─ error handling: on update failure, dump conv.to_dict() + failing batch + to an errorconv JSON in a dump dir, requeue once (retry cap), + then park the zid with a loud log (circuit breaker) +``` + +Config (mirrors Clojure + delphi's existing unwired poller config, +`delphi/polismath/components/config.py:216-269`): `DATABASE_URL`, `MATH_ENV` (the +math_env string written), `VOTE_POLLING_INTERVAL` (ms, default 1000), +`MOD_POLLING_INTERVAL` (1000), `POLL_FROM_DAYS_AGO` (10), `MATH_ZID_ALLOWLIST` / +`MATH_ZID_BLOCKLIST`, `POLISMATH_ENGINE_MODE`, worker-pool size. + +## 4. Cutover phases + +1. **Shadow (this PR):** new compose service `delphi-math-poller` (profile-gated, + `--profile delphi-math`) running next to the Clojure `math` service, writing under a + DIFFERENT `math_env` (e.g. `MATH_ENV=delphi` while Clojure writes `prod`/`dev`). + `UNIQUE(zid, math_env)` makes the rows invisible to the prod server. No consumer + change, zero production risk. +2. **Parity monitoring:** a comparer job diffs Python-vs-Clojure `math_main` rows per + zid/tick (ClojureComparer/stepcompare tolerance classes; R1 certification via the + replay harness feeds the same verdict). Exit criterion: agreed tolerance classes + green over an agreed soak window on dev + prodclone traffic. +3. **Flip:** point the server at the Python rows — either set the poller's `MATH_ENV` + to the server's `Config.mathEnv` (and stop Clojure), or flip the server's + `MATH_ENV`. One env-var change, instantly reversible. +4. **Decommission:** remove the `math` service from compose/deploy; archive the Clojure + tree (it remains the R1 oracle in the repo). + +## 5. Explicitly scoped out of phase 1 + +CSV export + correlation tasks (dead/covered — §2); `math_profile`; multi-worker +horizontal scaling (per-zid serialization makes a single instance correct; scale-out +needs zid sharding — later); DynamoDB writes (the existing delphi job pipeline is +untouched); storage-v2 provenance wiring (lands with that stack — the writer keeps a +seam for job-id/run-manifest fields). + +## 6. Testing + +- Unit: watermark advancement (strict `>`, max-of-batch), per-zid coalescing order + (votes before moderation, batch merge), writer SQL (caching_tick MAX+1, atomic tick) + against mocked cursors; bidToPid derivation from base_clusters; load-or-init + restoration path with a canned math_main blob. +- Integration (opt-in, needs Postgres like tests/test_postgres_real_data.py): end-to-end + poll→compute→write on a seeded conversation; shadow-mode row invisibility + (`math_env` isolation); restart resumes from math_main. +- Live shadow soak on dev compose = phase 2. diff --git a/delphi/docs/PLAN_DISCREPANCY_FIXES.md b/delphi/docs/PLAN_DISCREPANCY_FIXES.md index 8bb188bd9..bb2bf37aa 100644 --- a/delphi/docs/PLAN_DISCREPANCY_FIXES.md +++ b/delphi/docs/PLAN_DISCREPANCY_FIXES.md @@ -526,13 +526,13 @@ See `delphi/docs/INVESTIGATION_K_DIVERGENCE.md` for the full investigation. | ID | Discrepancy | Plan PR | GitHub PR | Status | |----|-------------|---------|-----------|--------| -| D1 | PCA sign flips | PR 13 | — (WIP) | VM draft — **NEEDS REWORK** (no replay tests) | +| D1 | PCA sign flips | PR 13 | — (spr-stack) | **RESOLVED for legacy mode (2026-07-18)**: sign stability delivered by the PCA warm-start chain (PR-B, `POLISMATH_ENGINE_MODE=clojure-legacy` threads prev comps as powerit start_vectors — Clojure's own mechanism; it has no explicit alignment either). Validated on real data: vw uniform-8 replay — improved (cold) mode flips PC2 at step 6, legacy holds orientation; legacy bit-deterministic across runs. Improved-mode cross-restart alignment deferred (needs persisted prev comps — design note in journal 2026-07-18). VM draft branch no longer exists anywhere; superseded by `SEQUENTIAL_BITS_PORT_SPEC.md`. | | D1b | Projection input | PR 13 | — (spr-stack) | **CODE FIX DONE ✓ (2026-07-17)** — `pca_project_cmnts` projected the untranslated Clojure literal `-1` (`-scale*(1+center)`) instead of the Delphi `AGREE` constant, INVERTING comment extremity (near-unanimous-agree → maximally extreme). Fixed to `scale*(AGREE-center)`. 3 tests (formula derived from AGREE, agree/disagree sign, extremity→priority_metric spy). Output-inert today (masked by #2571 `priority_metric` short-circuit) → no golden movement. Unblocks the D12 un-mirror. **Distinct from D1** (align_pca_signs / temporal stability). | | D2 | In-conv threshold | **PR 1** | **#2513** | **DONE** ✓ | | D2b | Base-cluster sort order | **PR 1** | **#2513** | **DONE** ✓ | | D2c | Vote count source (raw vs filtered matrix) | **PR 1** | **#2513** | **DONE** ✓ | | D2d | In-conv monotonicity (once in, always in) | **PR 1** | **#2513** | **DONE** ✓ (5 guard tests, T1-T5) | -| D3 | K-smoother buffer | PR 10 | — (WIP) | VM draft — **NEEDS REWORK** (no replay tests) | +| D3 | K-smoother buffer | PR 10 | — (spr-stack) | **GROUP LEVEL DONE (2026-07-18)**: group-k-smoother (buffer=4 + #2536 clamp + Clojure max-key higher-k-wins tie-break) landed as a pure function behind `clojure-legacy` mode; incremental no-flicker validated via chained update_votes. Base-cluster lineage + per-k warm start landed as **PR-C (#2622)**; in-conv greedy carry landed as **PR-E (#2623)**, both 2026-07-18. Subgroup smoother latent (Python has no subgroups) — NB the #2575 subgroup clamp (PR **#2609**) **MERGED to Clojure HEAD 2026-07-18**; the pre-merge "unclamped" target now applies only to a port certified against a pinned pre-#2609 ref. See `SEQUENTIAL_BITS_PORT_SPEC.md`. | | D4 | Pseudocount formula | **PR 2** | **#2514** | **DONE** ✓ | | D5 | Proportion test | **PR 4** | **#2519** | **DONE** ✓ (formula + n=0 short-circuit removed scalar/vectorized/caller — audit-discovered 2026-06-09, landed same day) | | D6 | Two-proportion test | **PR 5** | **#2520** | **DONE** ✓ | @@ -546,7 +546,7 @@ See `delphi/docs/INVESTIGATION_K_DIVERGENCE.md` for the full investigation. | D14 | Large conv optimization | — | — | **Deferred** (Python fast enough) | | D15 | Moderation handling | PR 12 | **#2523** | **DONE** ✓ (zero-out-columns + downstream `to_math_blob` / `_compute_vote_stats` regressions fixed 2026-06-09 — `to_dict` now routes through `_compute_user_vote_counts()` / `_compute_votes_base()`; `_compute_vote_stats` uses `_get_clean_matrix(raw=True)`) | | K-inv | Cold-start k divergence (row ordering) | (after D15) | **#2524** | **DONE** ✓ (FLI residual: inherent PCA divergence) | -| Replay | Replay infrastructure (A/B/C) | — | — | NOT BUILT — VM avoided this. D3/D1 used synthetic tests only. Needed for incremental blob comparison. | +| Replay | Replay infrastructure (A/B/C) | — | — (spr-stack) | **H-A BUILT (2026-07-18)**: schedule spec+slicer, Python driver, recording store + provenance, step comparer, CLI (`polismath/replay/`, `tests/replay_harness/`, 46 tests). Deterministic modulo `math_tick`. H-B (Clojure Mode A driver) **DONE 2026-07-18 (#2621)**. See `REPLAY_HARNESS_DESIGN.md` §11. | ### Non-discrepancy PRs in the stack @@ -937,22 +937,14 @@ Tagging this as a follow-up. No code changes until we discuss. - **`to_dynamo_dict` parallel inline implementations** were refactored to route through the same helpers as `to_dict` in PR #2523 follow-up. No further action needed. -- **`ns` includes-PASS-divergence** (DISCOVERED 2026-06-11 during D11). Clojure's - `:ns` (via `count-votes` with `filter identity` — repness.clj:56-61) INCLUDES - PASS votes. Python's `compute_group_comment_stats_df` and `consensus_stats_df` - both compute `ns = na + nd`, excluding PASS. This is a real divergence that - affects `pa, pd, pat, pdt, ra, rd, rat, rdt` everywhere — every downstream - metric and selection. The D5 PR #2519 journal claim ("PASS NOT included, - matching Clojure") was based on a misreading of `count-votes`. Currently - causing 3-5% divergence in pat values for tids with non-zero PASS counts; - visible at the consensus-selection margins (4/6 overlap on vw cold_start - agree, 1/3 on disagree). Needs a dedicated PR — affects: - - `compute_group_comment_stats_df` (line ~283: `ns = na + nd`). - - `consensus_stats_df` (line ~435: `ns = na + nd`). - Fix: `ns = (vote_matrix_df != 0).sum(axis=0)` no, actually we want to - count non-NaN: `ns = vote_matrix_df.notna().sum(axis=0)` for wide format; - for long-format `votes_long.groupby('comment').size()` after dropna. - Re-record goldens afterward. +- **`ns` includes-PASS-divergence** — **RESOLVED 2026-06-11** (landed with the + parity stack; this entry was stale until 2026-07-18). Both production + functions now count PASS-inclusive non-nil votes matching Clojure's + `count-votes` (repness.clj:56-61): `compute_group_comment_stats_df` uses + `ns=('vote','size')` over non-nil long-format rows (repness.py:258-266) and + `consensus_stats_df` uses `vote_matrix_df.notna().sum(axis=0)` + (repness.py:675); the `prop_test_vectorized` docstring (repness.py:94-99) + documents the PASS-inclusive contract. - **D10 take-5 eviction edge case** (2026-06-11). The Clojure-parity `select_rep_comments_df` introduced in PR 8 mirrors Clojure exactly: `take(5)` runs AFTER prepending the `best_agree` slot. When `best_agree` diff --git a/delphi/docs/SEQUENTIAL_BITS_PORT_SPEC.md b/delphi/docs/SEQUENTIAL_BITS_PORT_SPEC.md new file mode 100644 index 000000000..4750cde4f --- /dev/null +++ b/delphi/docs/SEQUENTIAL_BITS_PORT_SPEC.md @@ -0,0 +1,158 @@ +# Sequential-Bits Port Spec — Clojure warm-start/stateful behaviors → Python `clojure-legacy` mode + +**Status:** Verified inventory + port design — 2026-07-18 +**Provenance:** Produced by a read-only spec agent (opus) during the overnight orchestration +session of 2026-07-17→18; every file:line reference originally verified against the working +tree at commit `c51b7425` (D1b tip). Integrated and reviewed by the session integrator. +**Citations re-verified against commit `2d6c87035` (2026-07-18)** — conversation.py grew +~250 lines since `c51b7425`, so the conversation.py:NNN cites below were re-grepped and +corrected (rows 2, 3, 7, 11, §2.4, and the subgroup headline bullet). +**Companions:** `REPLAY_HARNESS_DESIGN.md` (the validation substrate), +`PLAN_DISCREPANCY_FIXES.md` (D1/D3 entries), `CLJ-PARITY-FIXES-JOURNAL.md` (session log). + +Python's delphi engine does full cold recompute every tick; Clojure threads warm-start +state across `conv-update` ticks. This spec is the complete inventory of that state and +the plan for reproducing it in Python behind an engine-mode flag, so that +Python-legacy-mode can replay Clojure trajectories step-for-step (R1/R2 prerequisite). + +## Headline verification results + +- **Group-level stale-k clamp (#2536): PRESENT** in Clojure HEAD — + `math/src/polismath/math/conversation.clj:469-478`. +- **Subgroup-level stale-k clamp (#2575): NOW IN Clojure HEAD.** PR #2609 (`207fa9f93`, + branch `jc/subgroup-k-smoother-clamp`) **merged 2026-07-18T02:52**, so current production + Clojure now runs the **clamped** subgroup smoother (`conversation.clj:534-559`) with its + regression test present in `math/test/conv_edge_cases_test.clj`. HISTORICAL NOTE: the + overnight port work targeted PRE-merge HEAD, where the subgroup smoother was still + UNCLAMPED — the "unclamped target" therefore applies only to a port certified against a + pre-#2609 frozen Clojure ref (pin the edge SHA). → keep it a sub-flag keyed to the + Clojure version being certified against. +- **Uncatalogued sequential divergence:** Clojure's `:comment-priorities` node reads the + **previous tick's** group-votes (`(:group-votes conv)`, `conversation.clj:650`), not the + freshly computed one. Python uses the current tick's. Currently masked by the #2571 + all-49 bug-mirror; must be honored when the mirror is removed. +- **Python computes no subgroups at all** (`conversation.py:1048` hardcodes + `subgroup_clusters = {}`), so all subgroup-level behaviors are latent port items. + +## 1. Inventory — cross-tick stateful behaviors + +| # | Behavior | Clojure (reads prev conv) | In `math_main`? (prep-main whitelist, conv_man.clj:52-74) | Python status | +|---|----------|---------------------------|------------------------------------------------------------|---------------| +| 1 | PCA warm start (small conv) | conversation.clj:381-387 `:start-vectors (get-in conv [:pca :comps])` | `:pca` YES | **GAP** — powerit_pca has `start_vectors` (pca.py:169-237) but no production caller passes it | +| 1b | PCA warm start (large conv, mini-batch partial-pca) | conversation.clj:755-765 | `:pca` YES | GAP, deferred (D14; no large-conv path in Python) | +| 2 | Base-cluster warm start | conversation.clj:403-410 `:last-clusters (:base-clusters conv)` → clean-start-clusters | `:base-clusters` YES | base k-means block (conversation.py:842-888); legacy warm start now ported (PR-C #2622) | +| 3 | Group-clustering warm start (per k) | conversation.clj:433-445 `((:group-clusterings conv) k)` | NO | per-k group loop (conversation.py:996 improved / 952 legacy-warm); legacy warm start now ported (PR-C #2622) | +| 4 | group-k-smoother (buffer=4 + #2536 clamp) | conversation.clj:454-478 | NO | **GAP** — best-k picked fresh each tick, no memory | +| 5 | Subgroup-clustering warm start | conversation.clj:492-520 | NO | GAP (latent — no subgroups) | +| 6 | subgroup-k-smoother (buffer=4, **clamped** in HEAD since #2609 merged 2026-07-18) | conversation.clj:534-559 | NO | GAP (latent) | +| 7 | comment-priorities ← prev-tick group-votes | conversation.clj:650 | `:group-votes` YES | **DIVERGENT** — Python uses current tick (conversation.py:1366, group_votes recomputed at :1423); masked by #2571 mirror | +| 8 | in-conv monotonic carry + greedy top-15 | conversation.clj:243-269 | `:in-conv` YES | threshold part EQUIVALENT via full recompute (append-only votes); **greedy top-15 DIVERGENT** — Clojure persists greedily-added pids forever, Python recomputes them each tick | +| 9 | customs pids/tids caps (100k/10k) | conversation.clj:167-189 | partial | DIVERGENT edge — no caps in Python; only matters above thresholds | +| 10 | last-vote-timestamp monotonic max | conversation.clj:161-165 | YES | EQUIVALENT | +| 11 | mod-out/mod-in/meta-tids carry | conversation.clj:838-876 (incremental conj/disj per event) | YES | ~EQUIVALENT; Python replaces whole sets per payload (conversation.py:598-643, sets replaced at :621-626) — mid-conversation un-moderation trajectories can diverge; replay slicer must feed the event stream | + +## 2. Key semantics (binding for the port) + +### 2.1 PCA warm start +`powerit-pca` (pca.clj:86-105): per-component power iteration, FIXED iteration budget +(default 100, `:pca-iters`) identical cold vs warm; start vector = previous tick's +post-normalization unit component; new columns absorbed by 1-padding (pca.clj:46-49); +all-zero start → re-randomized (`wrapped-pca`, pca.clj:108-124). Python's `powerit_pca` +already implements all of this faithfully (1-padding pca.py:120-123, zero→None +pca.py:224-227, exact-equality early exit pca.py:131-144) — the port is pure plumbing: +store previous `pca['comps']`, pass as `start_vectors`. Cold first tick: Clojure uses +unseeded `(rand)`; Python uses deterministic seed-42 (documented deliberate divergence, +pca.py:71-86). Warm start applies only to the powerit impl (sklearn cannot inject start +vectors — REPLAY_HARNESS_DESIGN.md §12.6). + +### 2.2 group-k-smoother (buffer = 4, conversation.clj:454-478) +State `{last_k, last_k_count (default 0), smoothed_k}`: + +``` +this_k = argmax_k silhouette[k] # TIE-BREAK: Clojure max-key keeps the LATER + # arg over ascending keys ⇒ HIGHER k wins ties + # (current Python best_k loop keeps LOWER k) +same = last_k is not None and this_k == last_k +this_k_count = last_k_count + 1 if same else 1 +smoothed_k = this_k if this_k_count >= 4 else (smoothed_k ?? this_k) +smoothed_k = smoothed_k if smoothed_k in group_clusterings else this_k # #2536 clamp +state' = {last_k: this_k, last_k_count: this_k_count, smoothed_k} +``` + +It is "best k must win 4 consecutive ticks before taking over", NOT a ring buffer. +First tick: smoothed_k None → accepts this_k ⇒ cold-start behavior unchanged (hard +regression constraint). Clamp is independent of the buffer: protects against the carried +k falling out of range when `max-k-fn = min(5, 2 + floor(n_base/12))` shrinks. + +### 2.3 Base-cluster lineage (clean-start-clusters, clusters.clj:230-277) +1. `safe-recenter-clusters` (171-191): recenter each existing cluster on its surviving + members; DROP clusters whose members all vanished; if all vanish → one fallback + cluster with fresh id `(inc max-id)`. +2. `uniqify-clusters` (220-227): merge identical-center clusters; merged cluster keeps + the LARGER side's id (194-199); center = size-weighted mean. +3. Split loop: while `min(k, #distinct rows) > count(clusters)`: pull the most-distal + point (202-217) into a NEW cluster with id `(inc max-id)`; recenter; repeat. + +Cluster ids are stable across ticks; new ids strictly increase. Cold init +(`init-clusters`, 55-65): first k distinct rows in encounter order, ids 0..k-1. +Iteration: `cluster-step` until `same-clustering?` (sorted-center distance < 0.01, +clusters.clj:68-76) or max-iters (base level uses `:base-iters` = 100). +**Python's off-production `clusters.py` warm-start machinery (302-403) is a DIFFERENT +algorithm — do not reuse.** Legacy mode needs a faithful numpy port and must use the +ported k-means loop (not sklearn Lloyd) when warm-starting. + +### 2.4 in-conv greedy carry +`in-conv = carried ∪ {p : votes[p] >= min(7, n_cmts)}`; if |in-conv| < 15, greedily add +top-(15−n) by vote count. Clojure PERSISTS greedy admits (carried set). Python recomputes +greedy each tick → churn while <15 qualifiers. Threshold part is provably equivalent +under full recompute of append-only votes (see D2d design, conversation.py:1691-1751); +greedy part needs a carried set in legacy mode. + +### 2.5 Persistence / worker restart +Clojure persists only the prep-main whitelist (`:pca`, `:base-clusters`, +`:group-clusters`, `:in-conv`, `:group-votes`, ...) — NOT smoother state, NOT per-k +clusterings. Worker restart = partial cold start (smoother counters reset, per-k warm +start lost; PCA/base warm start survive via math_main). For R1/R2 (single-process +chains) Python needs NO new serialization — state threads in memory across +update_votes calls. Restart modeling can later be encoded as schedule input boundaries. + +## 3. Python design + +- Engine switch `POLISMATH_ENGINE_MODE ∈ {clojure-legacy, improved}`, default `improved` + (byte-identical to today — hard gate), resolved via the `_resolve_impl_flag` idiom + (pca.py:37-57). Sub-flag for the #2575 subgroup clamp when subgroups ever land. +- New Conversation fields (cold defaults): `group_clusterings={}`, + `group_k_smoother={}`, later `prev_group_votes={}`, persistent `in_conv=set()`. +- Legacy-mode recompute order: PCA(start_vectors=prev comps) → in-conv (union into + carried set) → base kmeans(last_clusters=prev base) → per-k group + kmeans(last_clusters=prev per-k) → smoother → group_clusters = clusterings[smoothed_k] + → priorities over prev_group_votes (once #2571 un-mirrored). +- No to_dict/to_dynamo_dict changes (D2d precedent: defer persistence until delta + processing or worker-restart modeling requires it). + +## 4. PR split + +| PR | Content | Size | Status | +|----|---------|------|--------| +| A | POLISMATH_ENGINE_MODE flag + prev-state scaffolding + cold-invariance tests | S | overnight 2026-07-18 | +| B | PCA warm start (legacy) — thread prev comps → start_vectors | S | overnight 2026-07-18 | +| D′ | group-k-smoother (buffer=4 + #2536 clamp + Clojure tie-break), smoother only | M | overnight 2026-07-18 | +| C | Clojure-exact base k-means lineage port + base & per-k warm start | L | landed 2026-07-18 (#2622) | +| E | in-conv greedy carry (legacy) | S | landed 2026-07-18 (#2623) | +| F | priorities ← prev-tick group-votes | S | blocked on #2571 un-mirror | +| G | subgroups + subgroup smoother (+#2575 sub-flag) | L | latent | +| H | large-conv partial-pca (D14) + optional persistence | L | deferred | + +## 5. Test plan (summary) + +Cold-start invariance gates first (improved == today byte-for-byte; legacy first tick == +improved first tick). Smoother units: buffer counting, reset-on-change, clamp, tie-break +direction, first-tick acceptance. PCA warm-start: chained two-tick spy test + angle +tolerance. Lineage tests (PR-C): vanish-drop, `(inc max-id)` policy, merge-keeps-larger-id, +stable ids across chained updates; port conv_edge_cases_test.clj:66-98 (clamp) and +:110-127 (agg-bucket unknown pid). Incremental k-stability via the replay harness once +H-A lands; full R1 schedule-CCR parity once H-B records Clojure CCRs. + +Expected xfail harvest (eventually): D3 xfails in test_discrepancy_fixes.py; several +incremental-variant divergences (gid/membership trajectory); the D12 priority-parity +blocker chain (extremity → D1/D1b → un-mirror).