Add Hubble BigQuery checkpoint snapshot source - #1996
Draft
leighmcculloch wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add an optional
hubblefeature tosoroban-ledger-snapshot-source-txthat resolves checkpoint ledger state from the Stellar Hubble BigQuery dataset, replacing the history-archive bucket download for the contract data and TTL entries it can represent exactly.Why
Resolving an entry untouched in the replayed ledger range currently downloads and linearly scans the entire bucket set at the enclosing checkpoint, which can be many gigabytes; Hubble supports random access by ledger key, turning that into a single point query.
Known limitations
Hubble stores raw XDR for contract data only, so just
ContractDataandTtlentries are served and everything else — includingContractCode, whose Wasm bytes Hubble omits entirely, classic entry types, which have no XDR column, and nonce-keyed contract data, which stellar-etl discards before loading — falls back to the history archive. Hubble's finest granularity is a whole ledger, which is sufficient here because transaction-granular state comes from replaying ledger-close meta and Hubble is consulted only for the ledger-granular checkpoint fallback. Only mainnet data is published, there is no network discriminator column, andLedgerEntryextension fields are unrecoverable soextis set toV0, matching the existing RPC source. Because Hubble lags the network, every lookup is gated on the table being read having ingested past the checkpoint, so a stale or not-yet-loaded row can never be cached as an answer. Enabling the feature can only change how fast an answer arrives, never whether it is correct: any error also falls back.SemVer Change
vX._._) - Breaking change to the public API.v_.Y._) - Additive change to the public API.v_._.Z) - No change to the public API.