Skip to content

feat(prover): persist the derived Lagrange SRS so it isn't re-derived on every start - #3733

Open
coeniebeyers wants to merge 31 commits into
LFDT-Lineth:mainfrom
coeniebeyers:upstream-3584-srs-persist
Open

feat(prover): persist the derived Lagrange SRS so it isn't re-derived on every start#3733
coeniebeyers wants to merge 31 commits into
LFDT-Lineth:mainfrom
coeniebeyers:upstream-3584-srs-persist

Conversation

@coeniebeyers

@coeniebeyers coeniebeyers commented Aug 6, 2026

Copy link
Copy Markdown

This PR implements issue(s) #3584.

Stacked on #3732: the first two commits are that PR's WriterstoEqual fix, which the new verifying-key check here depends on; this branch will be rebased once #3732 lands.

When no Lagrange-form SRS dump is on disk, SRSStore derives it from the canonical SRS in memory and discards it on exit — a multi-hour derivation repeated on every prover start, with a single debug-level log line as the only indication. This PR makes the derivation happen at most once per store directory, by separating reading from provisioning:

  • GetSRS (prove time) remains a pure read. A miss still derives in memory, but at real circuit sizes it now warns before the derivation and names the command that resolves it. It never writes: the SRS directory can be mounted read-only.
  • prover setup provisions. It derives, validates and atomically publishes the dump under the provenance tag derived — locally computed material never claims a ceremony's name — and backfills a missing dump even when circuit assets are current. persist_derived_srs defaults to on; set it to false if the SRS directory must never be written to (for example, a read-only or shared asset store). --force re-validates an existing dump in full and replaces it if it does not load.

Failure modes, each pinned by a test:

failure handled by
partial or interrupted write temp file + fsync + atomic rename — never published
write failure (full disk, read-only directory) probe fails before deriving; setup warns and continues
truncated or unparseable Lagrange dump ReadDump error → re-derive
wrong-size dump point-count check → early re-derive
dump from a different setup verifying-key equality against the canonical → re-derive
corrupted point data on-curve validation → re-derive
corrupt or missing canonical (ceremony) file remains fatal — never re-derived
crash mid-write (e.g. an out-of-memory kill) orphaned temp file deleted at the next prover setup run, once its last write is more than an hour old
concurrent index mutation RWMutex, snapshot reads
different user id on a later run dumps published 0644, matching the ceremony files

Behaviour changes:

  • An unloadable Lagrange dump previously failed GetSRS; it is now logged and re-derived. Canonical (ceremony) load failures remain fatal.
  • A file named kzg_srs_canonical_<n>_<curve>_derived.memdump — a name the code never produces — was previously accepted as trusted ceremony material; it is now warned about and ignored.
  • Orphaned temp files are reclaimed by prover setup rather than at store construction; reads never modify the directory.
  • The copy-prover-assets Makefile target now excludes *_derived*, keeping derived dumps machine-local rather than syncing them into the shared asset bucket. If distributing them through the bucket is preferred, remove the exclusion — we left that as an explicit decision for the maintainers.

Numbers: loading a Lagrange dump from disk is the pre-existing fast path — deployments that ship pre-derived dumps already pay it on every start (~14 s at 2^27 on a real 13 GiB bls12-377 dump, 8 cores: 9.3 s read, 4.8 s parse). What this PR changes is who can take that path: deployments without shipped dumps previously re-derived in memory for hours on every start; now prover setup writes the dump once and later starts load it like any other. The one added cost is ~1.3 s of load-time validation (point count, verifying-key equality, on-curve scan), since the store may now be loading locally-written rather than only ceremony-shipped material. Disk grows by one dump per Lagrange size, the same order of magnitude as the canonical files already present. The feature was also exercised end-to-end through the built CLI: every operator flow (persist, no-op re-run, backfill, --force repair, orphan cleanup, opt-out, read-only directory) behaves as documented, and at 2^21 a fresh process loads the persisted dump in 0.1 s against the 93 s derivation it replaces.

Substitution of validly-encoded points is out of scope, as it is for every file in the store. gofmt, golangci-lint and the test suite (-race, -tags nocorset,fuzzlight) all pass.

Checklist

  • I wrote new tests for my new core changes.
  • I have successfully ran tests, style checker and build against my new changes locally.
  • If this change is deployed to any environment (including Devnet), E2E test coverage exists or is included in this PR.
  • I have informed the team of any breaking changes if there are any.

No breaking API changes.

Potential follow-up issues

Found while implementing this change and exploring the codebase; none are addressed in this PR, and we are happy to file each as its own issue:

  • protocol/serde/decoder.go — an integer overflow can bypass the deserializer's length bounds check, so a corrupted or crafted asset can be accepted (the same pattern appears at two more sites in the file).
  • cmd/controller/controller.go — a crashed large-mode job is recovered under a file name no input pattern matches, silently dropping it from the queue.
  • lib/compressor/blob/v2/test_utils — buffer aliasing causes the multi-blob test to run on duplicated data rather than two distinct blobs.
  • symbolic/constructor_ext_test.go — the parent-structure assertion compares the expected value to itself, so its fourteen fixtures verify nothing.
  • circuits/aggregation/circuit_test.go — a memoized test SRS makes every "different" verifying key byte-identical, so per-key selection is never exercised.

Thanks

Thanks @mkrielza for your help shaping this PR and for finding the WriterstoEqual issue.

mkrielza and others added 30 commits August 6, 2026 10:34
Signed-off-by: Marelize <marelize.kriel@adhara.io>
Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…test

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
… on every start

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…ename

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…mment

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
Signed-off-by: Marelize <marelize.kriel@adhara.io>
…eremony material

Signed-off-by: Marelize <marelize.kriel@adhara.io>
Signed-off-by: Marelize <marelize.kriel@adhara.io>
…ault off

Signed-off-by: Marelize <marelize.kriel@adhara.io>
…g even when circuits are current

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…emory derivation

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
… when unwritable, and quiet at dummy sizes

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
… setup section

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…r at prove time

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…ling setup

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…of trusting it

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…after it

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…ence after setup

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…ation, not after

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…ts cleanup

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…rveFileNames

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
…ap over-limit lines

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
… not-found

Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
Signed-off-by: Coenie Beyers <coenie.beyers@adhara.io>
@coeniebeyers

Copy link
Copy Markdown
Author

Review notes — observations we deliberately did not act on in this PR:

  1. A self-generated canonical SRS cannot be named honestly. Canonical files only accept ceremony tags, so a self-hoster who generates their own canonical SRS (the scenario in [Prover] Persist the derived Lagrange SRS so it isn't re-derived on every start #3584) has to label it with a ceremony's name. This PR gives Lagrange dumps an honest derived tag and rejects the accidental canonical_*_derived combination; letting canonical files carry an honest non-ceremony tag would be a naming-scheme decision we did not want to make unilaterally. Happy to open an issue if there is interest.
  2. lagrangeSizeWarnThreshold is a var rather than a const, solely so tests can lower it and exercise the warning path without a million-point derivation. If a different test seam is preferred (a store field, a parameter), glad to change it.
  3. Temp-file naming caveat: any file in the store matching kzg_srs_* and containing .memdump.tmp is treated as a crash leftover and reclaimed once its last write is more than an hour old; operator backups should avoid that suffix.
  4. persist_derived_srs = false also disables the orphan-temp cleanup — intentional, since the flag's contract is that setup never modifies the SRS directory, and deleting a file is a modification. A crash leftover on an opted-out deployment requires manual removal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants