architecture: make JMLE optimization and MMLE runtime fail closed on Rust

[seonghobae]

Production numerical ownership contract

JMLE production optimization must have one numerical owner. Python may validate, pack/unpack parameters, manage deterministic restarts, evaluate the model objective/gradient callback, construct results and retain explicit reference/test formulas. Adam moments/update arithmetic, L-BFGS history/two-loop/curvature/line-search behavior, optimizer phase sequencing, and result-affecting convergence control for backend="rust" belong in mlsirm-core/PyO3. Missing/incompatible Rust capability must fail closed.

Protected-main state — 2026-08-12

Fresh protected main: c91ae210bc8e0340a5b0587cf5289b82a2e8ab92.

Protected main already contains the accepted Rust/PyO3 ownership migration for advertised JMLE optimizer modes:

• backend="auto" fails closed when the compiled Rust core is unavailable rather than silently selecting NumPy;
• the legacy plain unidimensional public MMLE NumPy production fallback is removed; and
• public JMLE backend="rust" routes Adam, L-BFGS and adam_lbfgs sequencing through compiled jmle_optimize* entrypoints rather than the legacy Python _adam/_lbfgs loops.

Historical #751/#744/#738/#760 head/check identities are lineage only and do not transfer to current work.

Current recovery/convergence lane

Draft PR #770 is the active scientific acceptance slice. At this update its exact head is 503564b57ecf7ba504557399a3343f6dd203601e, but it is stacked on predecessor branch codex/fast-judge-accepted-type-regression at the older snapshot 11291fd5f03cf24e403289b0466a7b633d0d3971 rather than the predecessor's current head.

#770 adds a Rust L-BFGS convergence correction plus deterministic public-path recovery evidence for adam, lbfgs, and adam_lbfgs using aligned discrimination/easiness/theta bias/MAE/RMSE and explicit convergence evidence. Its prior stacked-head CI/ClusterFuzzLite success is useful lineage but is not merge evidence after the predecessor moved.

The current #778 predecessor is the repository's baseline documentation/fit-statistics correction line. It must integrate or become otherwise stable first; no predecessor check/review transfers into a refreshed #770.

What remains before closure

1. Integrate/stabilize the current test(judge): close accepted-type and ownership gaps #778 predecessor under live repository policy, then reconcile only fix(jmle): recognize objective convergence and add recovery evidence #770's unique optimizer/recovery delta onto the resulting exact lineage without destructive rebase/force-push.
2. Regenerate every base-sensitive Python 3.12/3.14, Rust/PyO3, package/reinstall, GPU-existing/fuzz, Security/SAST and current-head review gate on one unchanged refreshed fix(jmle): recognize objective convergence and add recovery evidence #770 head/current live base.
3. Preserve the ownership sentinel proving backend="rust" reaches zero legacy Python _adam/_lbfgs calls for all advertised optimizer modes.
4. Preserve optimizer identities, objective/gradient estimand, seed/restart semantics, traces/statuses and fail-closed behavior unless a separate scientific correction is reviewed test-first.
5. Require realistic aligned true-parameter recovery for Adam, L-BFGS and adam_lbfgs; ordinary fit-suite GREEN or correlation alone is insufficient.
6. Treat the finite deterministic recovery thresholds as scoped evidence, not a universal accuracy guarantee. Expand scenario/Monte Carlo coverage before broader claims.
7. Benchmark before adding an end-to-end GPU optimizer path; existing GPU objective/gradient parity is not optimizer parity.
8. Keep canonical Architecture/TRD/ADR/UML/ERD/traceability synchronized only through the sole docs: close canonical documentation fitness gaps without parallel authority #621/test(judge): close accepted-type and ownership gaps #778 documentation authority (or its explicit successor).

Non-goals

• no new estimator/model family merely for ownership convergence;
• no SciPy or hidden Python optimizer fallback;
• no hosted service/database change;
• no weakening of recovery, security or package gates.

Closure

Close only when the refreshed optimizer-mode recovery/convergence slice is integrated on protected main with current package/security evidence and the scientific recovery evidence relevant to the public claim is accepted. The Rust/PyO3 source ownership migration itself is already integrated.

[seonghobae]

Fresh protected-main scope extension from python/fast_mlsirm/backend.py at f43a466b1441b27bd2ac4a71be8aa3e4f08c8e8f: the production-owner defect is broader than _fit_mmle alone. VALID_BACKENDS still exposes {numpy, rust, auto}; explicit backend="numpy" resolves to NumPy, and backend="auto" silently resolves to NumPy whenever fast_mlsirm._core is unavailable. That means ordinary public production execution can still select a Python numerical owner even after fixing one estimator-specific fallback.

Treat this as part of #626 rather than opening a duplicate issue. The implementation must add fail-first public resolver/config tests and make the production contract fail closed on missing Rust. If a NumPy implementation is retained for parity/research, move it behind an explicitly named non-production test/reference interface rather than a normal FitConfig/CLI production backend. Align README/CLI/config/doctoring/package acceptance so auto means device/runtime selection within the Rust implementation—not algorithm-owner substitution—and ordinary installed-wheel execution cannot silently cross the Rust/Python authority boundary. Preserve parity tests by calling the reference surface explicitly in test code. No public compatibility claim should be removed silently; document any intentional API deprecation/version transition in the canonical #604 line and changelog.

[seonghobae]

Fresh protected-main API-truth finding at a7be9eb8ca0b807364173f8471fc300e3cf9350f: FitConfig.validate() currently accepts estimator='em' and 'bayes' because VALID_ESTIMATORS = {'jmle','mmle','em','bayes'}, while public fit() later raises NotImplementedError and labels both values “reserved for a future milestone.” This is a production contract/stub smell adjacent to this issue's fit.py numerical-owner migration: configuration validation advertises values that cannot execute.

Do not widen an active source branch solely for this while #628 and other Rust/PyO3 migrations are in flight. When #626 starts from then-current protected main, add a fail-first public-boundary contract and choose one honest behavior:

• preferred until a real estimator exists: remove reserved values from the accepted production estimator vocabulary so FitConfig.validate() fails early with the same bounded/non-reflective configuration error family; or
• implement a separately specified real estimator only with its own scientific identification/recovery/uncertainty contract.

Do not retain a value as “valid” merely because runtime raises NotImplementedError. Align public docs/configuration vocabulary and changelog, and preserve Rust-first numerical ownership. This addition is API-contract hygiene inside #626's existing fit.py migration, not evidence that EM/Bayesian estimation is now planned for that bounded implementation.

[seonghobae]

Fresh protected-main scope delta at a7be9eb8ca0b807364173f8471fc300e3cf9350f: FitConfig.validate() still advertises and accepts estimator in {"em", "bayes"} through VALID_ESTIMATORS, while fit() later raises NotImplementedError for both as “reserved for a future milestone.” This is a public contract/stub gap adjacent to this issue’s fit-runtime ownership work.

When implementing #626, revalidate the estimator vocabulary and close this false-capability surface without silently inventing an estimator. The smallest acceptable contract is: until a production Rust-owned EM/Bayesian estimator with identification/recovery/uncertainty evidence actually exists, generic config validation must not report an unimplemented estimator value as valid. Either remove the reserved values from the accepted production vocabulary (with a clear migration/deprecation path if external callers exist) or introduce an explicitly non-production experimental namespace that cannot reach ordinary fit() as accepted production configuration. Add RED→GREEN tests proving unsupported estimator names fail at config validation rather than surviving validation and raising NotImplementedError later.

Do not widen #626 into implementing Bayesian estimation merely to preserve the placeholder strings. Canonical PRD/TRD/maturity docs should classify those estimators as PLANNED/unsupported until real Rust implementation and scientific recovery evidence exist.

[seonghobae]

Current implementation authority update after protected-main refresh: protected main is now 7b57f77747b9a9bd911660c76b1be0ca0dc2d145. Stale Draft #717 was closed unmerged after proving it had exactly one unique path, tests/test_jmle_rust_optimizer_ownership.py. That fail-first contract was preserved verbatim on clean-lineage Draft #738, head ab2d627ebd2fe60bca2365f93c8561739aaebbc8, created directly from current protected main. No #717 checks/reviews/approvals/mergeability evidence transfers. Treat #738 as the sole active implementation authority for the JMLE optimizer-ownership remainder of this issue. Exact-current CI/Security/SAST for #738 are being recreated from scratch; keep this issue open until Rust/PyO3 optimizer ownership, recovery/parity/resource evidence, and protected-main acceptance land.

[seonghobae]

Fresh continuation on current protected main a891f843e9fb1f67688d38da759e80abb1e5ca44: Draft #770 (test/jmle-optimizer-recovery-evidence-20260812, exact head c8998e1dfc4d93173a6fc23ec3bf2a6ba9083512) now carries the missing optimizer-mode recovery acceptance slice. It exercises public Rust-CPU JMLE for adam, lbfgs, and adam_lbfgs over the same deterministic correctly specified 2PL-generating sample and requires convergence, objective improvement, discrimination/easiness bias+MAE+RMSE, and standardized-theta MAE/RMSE. CI/SAST are currently queued, so this is active evidence only and does not close #626. If a mode fails, retain the metric-rich failure as empirical evidence and repair the smallest optimizer/configuration root cause rather than weakening or deleting the gate.

[seonghobae]

Current protected-main evidence reviewed while continuing the Rust ownership loop: already fails closed when is unavailable, and the repository has a current doctoring/ADR/CHANGELOG boundary for explicit NumPy parity use. The remaining product gap is narrower: make the parity surface structurally non-production rather than exposing through ordinary FitConfig/CLI vocabulary.\n\nThat gap stays queued behind current protected-head review/check acceptance; it must be solved with an explicit reference API, migration documentation, and parity tests, without weakening the Rust production owner or silently breaking existing callers.

[seonghobae]

Current protected-main evidence reviewed while continuing the Rust ownership loop: resolve_backend("auto") already fails closed when fast_mlsirm._core is unavailable, and the repository has a current doctoring/ADR/CHANGELOG boundary for explicit NumPy parity use. The remaining product gap is narrower: make the parity surface structurally non-production rather than exposing backend="numpy" through ordinary FitConfig/CLI vocabulary.\n\nThat gap stays queued behind current protected-head review/check acceptance; it must be solved with an explicit reference API, migration documentation, and parity tests, without weakening the Rust production owner or silently breaking existing callers.

[seonghobae]

Opened PR #1070 from current protected main for the remaining backend contract slice. It moves NumPy fitting behind explicit fast_mlsirm.fit_reference / fit --reference surfaces, restricts ordinary FitConfig and production CLI backend choices to Rust (rust/auto), preserves parity tests, and updates the ADR/doctoring/README/changelog. Current-head checks and review are required before normal auto-merge.