Add streaming-loadtest ledger backend for apply-load replay - #679
Open
aditya1702 wants to merge 11 commits into
Open
Add streaming-loadtest ledger backend for apply-load replay#679aditya1702 wants to merge 11 commits into
aditya1702 wants to merge 11 commits into
Conversation
aditya1702
force-pushed
the
replay-loadtest-backend-pr6
branch
from
August 7, 2026 18:25
8bf66b4 to
d5ec0ae
Compare
This was referenced Aug 13, 2026
Open
aditya1702
force-pushed
the
replay-loadtest-backend-pr6
branch
2 times, most recently
from
August 20, 2026 17:23
f2ae385 to
68d3825
Compare
A dev-only LEDGER_BACKEND_TYPE=streaming-loadtest that reads stream-framed LedgerCloseMeta from named pipes written by stellar-core apply-load (one FIFO per transaction profile), renumbers each stream onto the consumer's requested sequence with per-pipe diffs, merges the per-sequence frames into one mixed-traffic ledger via the SDK's loadtest.MergeLedgers, and stamps monotone wall-clock close times (apply-load emits closeTime 0). Renumbering makes both sides restartable without a database reset: a restarted apply-load resets to raw sequence 1 and is mapped onto the next requested ledger; a restarted consumer resumes from its cursor. Pacing via --loadtest-ledger-close-duration bounds the ledger rate, and FIFO backpressure throttles the generators to match. Because apply-load's benchmark mode publishes no history archive, this backend type skips the archive connection and the cursor-0 checkpoint bootstrap: ingestion starts from ledger 1 on an empty database and balance state accumulates from the ledger stream. Everything downstream of the backend (live ingest loop, processors, persistence) is unchanged, so a load test exercises the same code path as production ingestion. Includes an opt-in corpus test (STREAMING_LOADTEST_CORPUS) that replays real apply-load output through the backend and the production transaction reader; verified against v27 sac/custom_token/soroswap corpora. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Renumber only the ledger header. The per-entry rewrite of lastModifiedLedgerSeq/TTL/seqLedger was a full XDR marshal/parse/walk/re-marshal round trip over every frame, and nothing in wallet-backend reads those fields — measured at ~5x the entire ledger-processing cost at full per-ledger volume (the stream's cadence was bound by it), and a 37x speedup on the corpus replay test once removed. Merging now appends transaction-set phases and results directly instead of going through the SDK's renumbering merge, which also drops the goxdr dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…data Two fixes for protocol-framework behavior surfaced by the loadtest rig: Protocol state production (SEP-41 balances/state changes) is gated by per-protocol cursor rows that only the protocol-migrate CLI creates, and that CLI needs a replayable ledger source. A streaming-loadtest deployment has none, so a fresh database silently skipped every protocol processor. The archive-less bootstrap now seeds each registered protocol's cursors at startLedger-1 — exactly the value the first ledger's compare-and-swap expects — and refreshes the cursor snapshot so production is live from ledger 1. The SEP-41 metadata fetcher re-fetched failing contracts on every ledger they were active in: claimed contracts re-enter Prefetch each classification pass, and a persistently failing name() simulation costs its full retry-with-backoff (600ms of sleep) every time. A negative cache now skips recently failed contracts for 5 minutes, keeping eventual enrichment while removing the per-ledger tax. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prefetch runs for every claimed contract observed in a ledger's changes and has no database access by design, so it re-simulated name/symbol/decimals for tokens whose metadata was already persisted — on every ledger that touched their instance. An in-process success cache skips them; a restart refetches each contract once, which doubles as the refresh path for tokens whose on-chain metadata changed. Persist retries are unaffected: the fetched values live in the classification plan, which is reused across retry attempts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each pipe's reader now buffers two decoded frames beyond the one in flight, so a writer streams its next frames through the FIFO while the consumer merges and processes earlier ledgers. With lockstep delivery every GetLedger waited on the slowest writer's in-flight frame — a constant ~0.6s/ledger at full volume that the one-frame handoff could not hide. Backpressure still bounds the writers, now with three frames of slack; an epoch's terminating error stays ordered behind its frames, so restart handling is unchanged.
…allel apply-load's meta write is synchronous: core does not start generating its next ledger until the consumer drains the current frame, and the reader drained only as fast as it decoded — putting XDR decode inside every producer's ledger cycle (measured: cycle = generation + decode, which capped the merged stream well under target cadence). Each pipe's reader now slurps a record's raw bytes at transfer speed and hands them to a per-pipe decode goroutine, so the producer starts its next ledger while the previous frame decodes. Frame and error ordering are preserved: a drain or decode error is always the last element delivered and ends the epoch exactly as before.
The streaming-loadtest backend's meta sources generalize beyond FIFOs: a tcp-listen://HOST:PORT entry binds a listener eagerly at construction and serves each producer connection as one stream epoch, so apply-load producers can run in their own pods and dial in. Connection close is the epoch boundary (as FIFO EOF is), the listener outlives epochs to serve producer restarts, and keepalive surfaces a vanished peer as a read error. Frame draining, decoding, renumbering, and merging are shared between both source kinds; entry order still defines merge order. The flag is now --loadtest-meta-sources / LOADTEST_META_SOURCES.
Each buffered lookahead frame is a fully decoded LedgerCloseMeta — tens of MB of pointer-dense XDR per source, ~600MB resident across 12 sources per lookahead unit — and that mass is scanned by every GC cycle. One frame of lookahead still overlaps the writer's streaming with the consumer's processing; the second bought no cadence and cost scan time.
The metadata fetcher's fetched/failure caches are in-memory only, so a token whose contract_tokens row already carries metadata was still re-fetched over RPC — once per process lifetime after every restart on a real network, and forever on a 5-minute backoff against a deployment whose RPC can never resolve metadata (the loadtest rig's dead endpoint with externally seeded rows), where each retry burned the ~850ms simulate backoff ladder inside prepare_classification. After Apply persists a batch's rows, contracts whose row has metadata are marked fetched, making the database the durable cache. Tokens whose metadata is genuinely missing stay unmarked and keep the existing backoff retry.
apply-load pins its ledger protocol to the core binary's current version (neither LEDGER_PROTOCOL_VERSION nor its testing override is settable from a config file), and every protocol since 23 emits V2 meta, so the V1 arms in appendLedger/setLedgerSeq/stampCloseTime were unreachable. Collapse the header rewrites onto one mutableHeader accessor that owns the V2-only invariant, rebuild the test fixtures on V2 (the shape production actually sees), and pin explicit rejection of V0 and V1 streams.
The fetched/failedUntil caches sit on the production ingestion path but had no coverage: nothing verified that a resolved contract stops hitting the RPC, that a failed fetch is suppressed for the backoff window, or that expiry re-fetches and success clears the failure entry. Add those tests; shortening metadataFailureBackoff here is what makes its var-for-tests comment true.
aditya1702
force-pushed
the
replay-loadtest-backend-pr6
branch
from
August 20, 2026 17:31
68d3825 to
745220e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Start with
streaming_loadtest_ledger_backend.go(the backend commit), then the TCP-producers commit. Review time: ~40 min.Dev-only ledger backend that replays
stellar-core apply-loadmeta straight into live ingestion — the real ingest loop at rates no test network can produce. Only the ledger backend is swapped; processors and persistence are untouched, so what it measures predicts production.Based on
blend/pr6-integration-tests(what dev runs today); re-targets tomainwhen pr6 lands.What's in it
LEDGER_BACKEND_TYPE=streaming-loadtest: merges framed meta from N producer sources into one ledger sequence, renumbers so restarts never need a DB reset, stamps advancing close times, paces via--loadtest-ledger-close-duration. No history archive — starts from ledger 1 on an empty DB.--loadtest-meta-sourcestakes FIFO paths and/ortcp-listen://HOST:PORT, so producers run in their own pods and dial in. Connection close = epoch boundary (same as FIFO EOF); the listener outlives epochs so producers can restart.Review path
streaming_loadtest_ledger_backend.go— merge, renumber, pacing (the bulk)Testing
Unit tests: renumbering, multi-source merge, generator restart, truncated frames, pacing, TCP epoch lifetime —
-raceclean. Opt-in corpus test (STREAMING_LOADTEST_CORPUS) replays real v27 apply-load output: 200 merged ledgers, ~57k txs.Deploy note
The rig env must rename
LOADTEST_META_PIPE_PATHS→LOADTEST_META_SOURCESwith this image. The producer side (dialer shim, StatefulSets, jitter) lives in the kube repo.First of four PRs from the live-ingest loadtest campaign (replaces #682). Review order: this → #684 → #685; #686 is independent. Combined rig result: 18,458 tx/s, process p99 0.992 s (window p90 0.768 / p99 0.995); PRs not separately re-benchmarked.