refactor(storage): unify structured mutation preparation

[Jurij89]

Summary

• Introduce a deeply immutable, privately trusted structured-mutation snapshot that owns caller isolation, guarded/touched graph effects, and no-op classification while allowing unchanged in-process wrappers to reuse one identity.
• Move deferred operand-budget validation and backend update generation into a Storage-internal materializer, with non-normalizing builders so leaf adapters construct update text once without a second O(N) operand copy.
• Migrate Agent, GraphSet, Changelog, literal-blob, HTTP, Blazegraph, embedded Oxigraph, and worker-backed paths while preserving each owner's capability, reserved-graph, close/drain, lifecycle, and failure-recovery semantics.
• Validate deferred operand budgets for candidates and no-ops before suppressing valid backend/HTTP/worker work, and externalize oversized SWM literals before final bounded materialization.
• Trust only nominal local/worker pre-dispatch refusals; operational no-op failures and forged refusal codes still trigger Changelog reconciliation and GraphSet refresh.
• Add golden-byte, API-boundary, type-contract, full-composition, worker-lifecycle, request-count, and reproducible Node 22 resource gates.

Related

• Fixes refactor(storage): unify structured mutation preparation #2208
• Follow-up to refactor(storage): own structured mutation commit effects #2194 and refactor(storage): split structured mutation modules #2196
• Rebased onto feat(agent): add bounded profile reconciliation #2206 integration head dd078f0aa41c7c973b3fa1504050e1caff61db01
• Found during integrate(storage/agent): land reviewed mutation and RS-heal stack #2207 review
• Related to Reconnect-triggered background sync amplification can starve foreground catch-up #2052

Diagrams

Structured mutation preparation

Before:

sequenceDiagram
    participant Caller
    participant Agent
    participant Decorators
    participant Leaf
    participant Backend
    Caller->>Agent: mutable descriptor
    Agent->>Agent: derive effects
    Agent->>Decorators: original descriptor
    Decorators->>Decorators: normalize and derive effects again
    Decorators->>Leaf: copied descriptor
    Leaf->>Leaf: normalize, validate, build update
    Leaf->>Backend: dispatch update or no-op request

Loading

After:

sequenceDiagram
    participant Caller
    participant Agent
    participant Decorators
    participant Leaf
    participant Backend
    Caller->>Agent: mutable descriptor
    Agent->>Agent: capture immutable snapshot once
    Agent->>Decorators: trusted snapshot mutation
    Decorators->>Decorators: reuse snapshot identity
    Decorators->>Decorators: optional literal rewrite and recapture
    Decorators->>Leaf: prepared mutation
    Leaf->>Leaf: deferred validation and one materialization
    alt executable
        Leaf->>Backend: dispatch one update
    else valid no-op
        Leaf-->>Caller: complete after policy/lifecycle preflight
    end

Loading

Files changed

File	What
packages/storage/src/bounded-structured-mutation.ts	Add deep-readonly capture, exact single-read copying, effects/no-op observation, private trust, and identity reuse.
packages/storage/src/structured-mutation-materialization-internal.ts	Add the Storage-internal deferred validation and executable update boundary.
packages/storage/src/structured-mutation/*.ts	Co-locate canonical capture/normalization, graph semantics, deferred operands, non-normalizing builders, and private refusal identity.
packages/storage/src/adapters/*.ts	Materialize only at HTTP, Blazegraph, embedded Oxigraph, and worker leaves; preserve policy and lifecycle ordering.
packages/storage/src/{graph-set-index-store,changelog-store,shared-memory-literal-blob-store}.ts	Reuse prepared mutations and preserve owner-specific maintenance/failure behavior.
packages/agent/src/dkg-agent-base.ts	Capture before forwarding and consume prepared effects for success-only cache invalidation.
packages/storage/src/index.ts and pack-gate fixtures	Export only the read-only capture contract and prove materialization is unreachable from supported package subpaths.
packages/storage/scripts/benchmark-structured-mutation.mjs	Add immutable baseline/compare modes with fresh Node 22 child processes and 10% CPU/heap/RSS gates.
packages/storage/test/*structured-mutation* and owner tests	Pin golden bytes, trust/type boundaries, caller races, all-kind composition, no-op I/O, literal ordering, and worker terminal states.
packages/agent/test/replace-subject-agent-wrapper.test.ts	Cover all six kinds, exact options/effects, caller isolation, no-op behavior, and Promise rejection semantics.

Test plan

• pnpm --filter @origintrail-official/dkg-core build
• pnpm --filter @origintrail-official/dkg-storage build
• pnpm --filter @origintrail-official/dkg-storage typecheck:type-contracts
• pnpm --filter @origintrail-official/dkg-agent build
• pnpm --filter @origintrail-official/dkg-storage test:package-exports (all properties held; all mutants killed)
• Named Storage matrix excluding separately serialized worker files: 10 files, 318 tests passed
• test/oxigraph-worker-resilience.test.ts: 11 tests passed
• test/oxigraph-worker-respawn.test.ts: 12 tests passed
• Focused structured-mutation, worker, and recovery matrix: 7 files / 157 tests passed when serialized; worker resilience 11/11 and respawn 12/12 pass in isolation
• Full serialized Storage suite under concurrent host load (--testTimeout=15000): 67 files / 1,053 tests passed; 26 configured integration/oracle tests skipped
• Named Agent matrix: 2 files / 22 tests passed
• pnpm exec tsx scripts/check-managed-store-raw-channels.mjs (287 reviewed dynamic sites; no raw update or statically mutating query calls)
• git diff --check
• Node v22.22.0, x64, fixture digest 94aeae872655f5489f105a15c2d88c43912adfdcbb55fa3401e94d453fc850f3, 100,000 subjects, 10 warmups, 30 trials, 3 fresh processes
• Benchmark Storage-identical base dd078f0aa41c7c973b3fa1504050e1caff61db01 (same packages/storage tree as the a5c04c257 baseline artifact) -> f7cbf464c250f30d48888d9b4ca16c094b80a239: CPU -53.37%, retained heap +0.18%, max RSS -6.29%

The repository's literal node scripts/check-managed-store-raw-channels.mjs command cannot map the source module's .js specifier to sparql-lexer.ts under Node's built-in type stripping; the same checked-in script passes through the repository's installed tsx loader as recorded above.

[Jurij89]

@otReviewAgent Please run a convergence review against exact current head bd9851d97e1559d982e7e13feeba433ea7ba034d. The four prior threads have current-head replies and are resolved: typed deterministic budget refusals remain mutation-free, per-kind modules now own canonical capture/normalization/semantics/materialization, malformed snapshot fixtures cover the production path, and the private WeakMap boundary is retained to preserve the approved unchanged TripleStore contract. Please evaluate correctness, recovery classification, trust/API boundaries, resource bounds, test coverage, and merge readiness on this SHA.

[Jurij89]

@otReviewAgent Please run a convergence review against exact current head aea83829b5275724825f99c993d99654a9fb8a4d. All current-head findings now have evidence replies and every review thread is resolved. This head adds stable worker refusal categorization, one structured-mutation refusal classifier, discriminated snapshot typing without effect assertions, and direct multi-graph guard plus full worker-stack recovery coverage; the explicit per-kind ownership model is retained intentionally. Please evaluate correctness, worker protocol compatibility, recovery classification, trust/API boundaries, resource bounds, tests, and merge readiness on this SHA.

[Jurij89]

@otReviewAgent Please run a convergence review against exact current head 3e986c4b10882f036c700ee0c889ccf84aa7e364. Both findings from the prior review are addressed with evidence replies and resolved: worker no-ops now validate deferred budgets before the lifecycle/no-I/O return, and materialization uses explicit per-kind validation plus separate normalized update builders with no boolean mode. Exact-head builds, type contracts, package gates, focused worker/composition coverage, the full 1,052-test Storage suite, and the resource benchmark all pass. Please evaluate correctness, worker protocol/lifecycle behavior, resource bounds, tests, and merge readiness on this SHA.

[Jurij89]

@otReviewAgent Please run a final convergence review against exact current head f7cbf464c250f30d48888d9b4ca16c094b80a239 on base dd078f0aa41c7c973b3fa1504050e1caff61db01. All review threads now have exact-head evidence replies and are resolved. This head makes clean refusal classification explicit at the materialization/worker-preflight phase: bare late budget errors and forged public codes trigger Changelog reconciliation plus GraphSet refresh, while genuine parent/worker pre-dispatch refusals remain nominal, serialized, and mutation-free. The leaf-choreography and snapshot-aware rewrite suggestions are documented as nonblocking follow-ups because current backend policy ordering and captured identity are intentional and fully covered. Please evaluate correctness, recovery classification, worker protocol/lifecycle behavior, trust/API boundaries, resource bounds, tests, and merge readiness on this SHA.