feat(store): persist the account SMT forest in SQLite - #2333
Conversation
JereSalo
added
the
maintainer
|
One other thing that could be interesting to measure is memory usage with this new structure. |
Yeah that's a pretty good idea. I measured it and updated the page with the memory numbers. The usage of RAM is dramatically lower with this new approach. And the amazing thing is that even though disk usage is greater, there's not that much of a difference. It highlights the fact that we had in memory a lot of internal nodes that we can just recompute from disk when we need them, trading off a tiny bit of performance but gaining a huge amount of memory when we have a lot of accounts. |
| /// Returns the lineage identifier for an account's asset vault SMT. | ||
| pub fn vault_lineage_id(account_id: AccountId) -> LineageId { | ||
| let mut bytes = Vec::new(); | ||
| bytes.extend_from_slice(b"miden-client:vault"); |
There was a problem hiding this comment.
The miden-client prefix is arbitrary. We could pick another one or even leave just "vault". It just matches the name of the crate.
Let me clarify on this, because things have changed a bit since this comment. That's why we now do the same as is done in miden-crypto in the RocksDB implementation of the Of course, the consequence of this is that the file on disk is an order of magnitude bigger than the implementation that just stored the leaves, but it's still a reasonably low number, so it's an acceptable tradeoff. It's basically 768 MB at 5 × 50k, or 2.9GB with an account with 1M kv entries, but performance-wise a witness read went from ~400 ms / ~8.5 s (at 50k / 1M entries per account, since it had to rebuild the account's tree from the leaves) to ~0.1-0.2 ms (warm) flat at every size, which is basically at parity with the old fully-in-memory forest (~0.09 ms). |
| # TEMPORARY (issue #2315): replaces the crates.io miden-crypto 0.28.0 (and its workspace | ||
| # siblings) with the branch that opens the LargeSmtForest Backend API for external | ||
| # implementations. Remove once that change is released as miden-crypto 0.28.x. | ||
| [patch.crates-io] | ||
| miden-crypto = { branch = "client-pin-from-parts-0.28", git = "https://github.com/JereSalo/miden-vm" } | ||
| miden-crypto-derive = { branch = "client-pin-from-parts-0.28", git = "https://github.com/JereSalo/miden-vm" } | ||
| miden-field = { branch = "client-pin-from-parts-0.28", git = "https://github.com/JereSalo/miden-vm" } | ||
| miden-lifted-air = { branch = "client-pin-from-parts-0.28", git = "https://github.com/JereSalo/miden-vm" } | ||
| miden-lifted-stark = { branch = "client-pin-from-parts-0.28", git = "https://github.com/JereSalo/miden-vm" } | ||
| miden-serde-utils = { branch = "client-pin-from-parts-0.28", git = "https://github.com/JereSalo/miden-vm" } | ||
| miden-stark-transcript = { branch = "client-pin-from-parts-0.28", git = "https://github.com/JereSalo/miden-vm" } |
There was a problem hiding this comment.
Note that this PR will be mergeable once the change from this PR is released on crypto 0.28.1 and we start using that.
We should update Cargo.toml and Cargo.lock when the time comes.
…stence # Conflicts: # CHANGELOG.md
miden-crypto 0.28.1 is still unpublished, so the patch remains, but it now pins the official 0xMiden/miden-vm repo at the squash-merge of 0xMiden/miden-vm#3413 instead of a mutable personal-fork branch.
2 participants
The account SMT forest (one sparse Merkle tree per account vault and per storage map slot, used to serve asset and storage witnesses) previously lived only in memory. It was rebuilt from the SQL tables on every store open and cloned on every write so failures could roll back. This PR moves it onto miden-crypto's
LargeSmtForestwith a new SQLiteBackend, so trees persist across restarts and mutations are two-phase (compute, then apply) instead of clone-and-swap.The backend borrows the store's own rusqlite transaction, so forest writes commit or roll back atomically with the account-table writes and there is no separate state to reconcile. Trees are addressed by lineage IDs derived deterministically by hashing the account id (plus the slot name for maps) and stored in four new tables (per-tree metadata, tree entries, the tree's inner nodes packed as 8-level subtree blobs, and a monotonic version counter). Witness reads assemble their proof from one leaf plus the eight subtree blobs on its path and verify it against the stored root, the same layout as miden-crypto's RocksDB forest backend, so read cost is independent of the account's map size. Tree updates are computed the same way, the backend derives the forward and reverse mutation sets from the affected leaves and stored subtrees only (assembled with
MutationSet::from_partsfrom the miden-vm PR below) instead of rebuilding the account's tree in memory, with every touched path authenticated against the stored root before anything is written. Undo paths reconcile the affected lineages to the restored tables as new forward mutations, versions never rewind.Measured against
nextwith a synthetic store of 5 heavy accounts (50,000 map entries each), opening the store drops from ~32 s (linear in total state) to a roughly flat 0.2-0.3 ms, opening no longer loads the forest into RAM (~3 GiB of process growth before, ~1.6 MiB now), and committed updates are 9.8-15.8x faster at the sizes both builds ran (3.3 s to 0.27 s at 50k; the old design also deep-cloned the whole forest on every write as its rollback backup). Witness reads stay near the in-RAM forest's cost instead of scaling with the map (warm reads 0.10-0.16 ms from 1,000-entry maps to 1M). The trade-off moved to disk. Persisting the inner nodes makes the store file roughly 23-25x bigger than the leaves alone across the paired sizes tested (768 MB vs 33 MB for the 5 × 50k store). What still grows with the map is the store-side reconciliation,update_accountpresents the account's full snapshot on every update, so a one-entry change measures ~0.27 s at 50k entries and ~7.5 s at 1M, even though a dedicated timing harness that submits only the changed pair measures the forest-side update at ~1 ms; submitting only the changed pairs is a planned follow-up. Full benchmark results and methodology are here. The benchmarks are in the branch jere-smt-forest-bench.This depends on 0xMiden/miden-vm#3413, which opens the
BackendAPI for external implementations and needs to ship in a miden-crypto 0.28.x release. Until that happens,Cargo.tomltemporarily patches the crypto crates to a pinned branch on my fork carrying that PR's API additions on top of the 0.28.1 crypto (the PR branch itself tracks miden-vmnext, which has moved past the released protocol crates); the patch block must be removed (and the lockfile regenerated) before merging.Closes #2315. Closes #2256.