RFC-026: add private B2 token authority and exact fold recovery

[aaltshuler]

Summary

• advances the private RFC-026 implementation to internal schema v9, stream config v3, and lifecycle state v2
• adds canonical compare-and-chain payload/token digests, grammar-impossible trusted row metadata, same-generation overlays, and a manifest-selected graph-global _stream_tokens.lance authority
• folds base rows and token winners through exact pre-minted transactions owned by recovery-v12, with one __manifest CAS publishing both pointers, lifecycle state, lineage, and durable attribution
• recaptures lifecycle/binding/HEAD/token authority after shared admission and same-key queue ownership, and bounds B2 preprocessing to two 128 MiB envelopes
• adds the genuine v8↔v9 refusal/export-init-load rebuild gate while preserving logical rows, vectors, exact-id PK metadata, and ordinary __omnigraph_stream_v1 user data
• keeps the implementation crate-private and feature-gated: no production enrollment, lifecycle management, SDK, CLI, HTTP, Cedar, or OpenAPI surface is activated; B2a remains unbounded retain-all with no MemWAL GC
• realigns the existing branch-delete cost baseline with its current operation-local capture shape

Why

Private B1 proved durable admission and bounded folding, but it did not yet provide durable per-key sequencing, idempotent retry classification, trusted contributor attribution, or an exact two-participant fold outcome. This slice closes that correctness gap without exposing a product API or inventing an OmniGraph WAL/transaction manager beside Lance.

Compatibility and risk

This is an intentional internal storage-format boundary: v8 graphs are refused by v9 and must cross through export/init/load. Recovery-v11 remains historical and is refused under v9 because it lacks the token participant and complete lifecycle-v2 authority. The topology remains main-only, unsharded, and one live writer process; fresh reads and managed generation reclamation remain inactive.

Validation

• cargo test --workspace --locked
• cargo test --locked -p omnigraph-engine --features failpoints --test failpoints — 141 passed, 1 ignored
• cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream — 34 passed
• cargo test -p omnigraph-server --features aws --locked
• scripts/check-agents-md.sh
• git diff --check

Greptile Summary

The PR completes the private RFC-026 B2 compare-and-chain storage path.

• Adds canonical stream tokens, trusted row attribution, same-generation overlays, and graph-global token authority.
• Introduces recovery-v12 to atomically recover and publish base-table and token-authority fold effects.
• Advances internal storage/config/lifecycle formats and adds the v8-to-v9 export/rebuild compatibility gate.
• Adds bounded B2 preprocessing and regression coverage for same-key chains, ambiguous acknowledgements, recovery, and authority races.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure eligible for this follow-up review remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
crates/omnigraph/src/db/omnigraph/stream_ingest.rs	Implements bounded B2 admission, same-generation token overlays, attributed winner folding, and joint base/token publication without an eligible unresolved defect.
crates/omnigraph/src/db/manifest/recovery.rs	Adds recovery-v12 validation and completion for the exact two-participant stream-fold outcome.
crates/omnigraph/src/db/manifest/stream_token.rs	Defines canonical payload/token digests, admission classification, trusted metadata, and fold-attribution commitments.
crates/omnigraph/src/db/manifest/token_store.rs	Adds bounded lookup, staging, and validation for the graph-global stream-token authority.
crates/omnigraph/tests/memwal_stream.rs	Covers same-generation chaining, exact retries, ambiguous acknowledgements, authority recapture, and fold/recovery behavior.
crates/omnigraph-cli/tests/crossversion_upgrade.rs	Exercises the intentional v8-to-v9 refusal and export/init/load rebuild boundary.

Sequence Diagram

sequenceDiagram
  participant Caller
  participant Admission
  participant WAL as Lance MemWAL
  participant Base as Base table
  participant Tokens as Token authority
  participant Manifest
  Caller->>Admission: row + predecessor token
  Admission->>Admission: classify against durable/overlay authority
  Admission->>WAL: append attributed row
  WAL-->>Caller: durable acknowledgement
  Admission->>WAL: seal and scan latest winners
  Admission->>Base: commit pre-minted fold transaction
  Admission->>Tokens: commit pre-minted token transaction
  Admission->>Manifest: one CAS publishes both pointers

Loading

_{Reviews (2): Last reviewed commit: "test: pin the armed-before-any-effect St..." | Re-trigger Greptile}

Context used:

• Knowledge Base — OmniGraph core storage: manifest, table store, and Lance access
• Knowledge Base — OmniGraph write path