Skip to content

Add Hubble BigQuery checkpoint snapshot source - #1996

Draft
leighmcculloch wants to merge 1 commit into
snapshot-source-txfrom
snapshot-source-hubble
Draft

Add Hubble BigQuery checkpoint snapshot source#1996
leighmcculloch wants to merge 1 commit into
snapshot-source-txfrom
snapshot-source-hubble

Conversation

@leighmcculloch

Copy link
Copy Markdown
Member

What

Add an optional hubble feature to soroban-ledger-snapshot-source-tx that resolves checkpoint ledger state from the Stellar Hubble BigQuery dataset, replacing the history-archive bucket download for the contract data and TTL entries it can represent exactly.

Why

Resolving an entry untouched in the replayed ledger range currently downloads and linearly scans the entire bucket set at the enclosing checkpoint, which can be many gigabytes; Hubble supports random access by ledger key, turning that into a single point query.

Known limitations

Hubble stores raw XDR for contract data only, so just ContractData and Ttl entries are served and everything else — including ContractCode, whose Wasm bytes Hubble omits entirely, classic entry types, which have no XDR column, and nonce-keyed contract data, which stellar-etl discards before loading — falls back to the history archive. Hubble's finest granularity is a whole ledger, which is sufficient here because transaction-granular state comes from replaying ledger-close meta and Hubble is consulted only for the ledger-granular checkpoint fallback. Only mainnet data is published, there is no network discriminator column, and LedgerEntry extension fields are unrecoverable so ext is set to V0, matching the existing RPC source. Because Hubble lags the network, every lookup is gated on the table being read having ingested past the checkpoint, so a stale or not-yet-loaded row can never be cached as an answer. Enabling the feature can only change how fast an answer arrives, never whether it is correct: any error also falls back.

SemVer Change

  • Major (vX._._) - Breaking change to the public API.
  • Minor (v_.Y._) - Additive change to the public API.
  • Patch (v_._.Z) - No change to the public API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant