Fix the DB query-duration buckets, the oldest-ledger index path, and compression scheduling - #686
Open
aditya1702 wants to merge 96 commits into
Open
Fix the DB query-duration buckets, the oldest-ledger index path, and compression scheduling#686aditya1702 wants to merge 96 commits into
aditya1702 wants to merge 96 commits into
Conversation
A concurrent protocol-migrate engine snapshots membership from committed protocol_contracts rows only, so when it wins a ledger's cursor swap it has folded that ledger without contracts whose classification commits with live's in-flight transaction — their deploy-ledger state (constructor ContractData writes, first events) was extracted by nobody, and cursor-passed ledgers are never replayed. Never-rewritten keys stayed missing forever. On a lost swap where this ledger classified new contracts for the protocol and the lost cursor's committed value is at or past this ledger (a lost CAS blocked on the winner's row lock, so that value is reliably visible in-transaction via the new IngestStore.GetInTx), live now re-stages scoped to exactly the gap contracts and persists the lost halves in the same transaction, without moving any cursor. The CAS-winning path is extracted into stageAndPersistProtocolLedger alongside the new repairClassificationGap. Covered by CAS-gating cases M1-M4: frontier repair, behind-tip non-repair, partial-loss scoped repair, and already-committed exclusion.
…r is fetched The engine refreshed a tracker's classified-contract membership right after each window commit — milliseconds before a concurrent live transaction for that same contested ledger finishes committing a new contract's classification, since live's lost CAS was blocked behind the engine's row lock and still has work to do after unblocking. The engine then staged the next window with a snapshot missing that contract, skipping its events and entries for one more ledger; additive fold columns (cost basis, lifetime claimed totals) never heal from a missed ledger, so those deltas were permanently lost whenever the engine also won that next ledger. The refresh now runs at window start, after GetLedger returns for the window's first ledger: the fetch blocks until that ledger has closed, which is a full ledger interval after any concurrent transaction for the previous one committed. Same cadence — once per window per requiring tracker — with the read taken at the latest useful moment. The refresh test now pins the ordering: the first folded ledger must already carry membership committed after the run-start snapshot (mutation-verified — removing the window-start refresh fails it).
…sons Blend state changes follow the core convention: category names the on-chain object (BLEND_SUPPLY, BLEND_COLLATERAL, BLEND_DEBT, BLEND_AUCTION, BLEND_EMISSIONS, BLEND_BACKSTOP_EMISSIONS, BLEND_BACKSTOP, BLEND_BACKSTOP_QUEUE) and reason names the action. Amount-bearing categories reuse the generic CREDIT/DEBIT/ADD/REMOVE/ BURN verbs; only BORROW, REPAY, FLASH_LOAN, BAD_DEBT, FILL, and CLAIM are added.
…keys Postgres UPDATE ... FROM applies only one matching source row per target row, so duplicate (pool, user, asset) keys in a batch silently dropped deltas. Net deltas reject duplicates (ZeroBorrowed makes merging order-dependent; the processor pre-aggregates); auction adjustments are purely additive and are summed server-side before applying.
…odels Add blend_pool_claimed (pool, user) and blend_backstop_claimed (user) tables plus PoolClaimedModel/BackstopClaimedModel with additive BatchApplyDeltas. These hold lifetime claimed BLND / Comet LP totals, folded from claim events during current-state indexing — the only pass that sees every claim since Blend's first ledger. Mirrors the net_supplied/net_borrowed cost-basis accumulator.
…account_id Both tables are read by GetByAccount (the per-user positions path) filtering on user_account_id, which is the second PK column and so cannot use the primary key. Add single-column B-tree indexes mirroring idx_blend_positions_user, keeping the index defined alongside its table in the same migration.
ApplyAuctionAdjustments converted protocol tokens to underlying with exact numeric division, leaving a fractional tail (e.g. "1100.0000000000000000") in net_supplied/net_borrowed while every other write to those columns stores floored integer text. The contract uses fixed_mul_floor (floor of the positive magnitude). Wrap the conversion in trunc(): for the signed lot/bid deltas, trunc toward zero reproduces floor-of-magnitude-with-sign (trunc(-366.3) = -366, not floor's -367). Adds a subtest covering a fractional product on both the positive (filler) and negative (liquidated user) sides.
… writers BatchApplyNetDeltas and ApplyAuctionAdjustments mutate existing rows only; a delta for a not-yet-inserted position row silently no-ops. Document that callers must upsert the Positions snapshot (and reserves) first, as PersistCurrentState does.
The full-snapshot upsert/zero writers overwrote last_modified_ledger outright while the additive writers (net deltas, claimed, reserve data, reward zone) already use GREATEST. Switch the snapshot writers to GREATEST(<table>.last_modified_ledger, EXCLUDED/u.ledger) too, so the column never moves backward and every writer treats it uniformly. Behavior is unchanged under the strictly ledger-ordered persist path.
StartBlock and LastModifiedLedger were int32 while every other blend row struct uses uint32 for ledger-valued fields, casting to int32 only at the write boundary. Align Auction with that convention.
…ments Flooring doesn't distribute over addition, so n fills folded into one (pool, user, asset) row overstate magnitude by at most n-1 stroops versus per-fill flooring. The existing duplicate-key test uses amounts exact at the fixture rates, so it cannot tell the two orders apart; this one can (two fills of 5 at b_rate 1.1: 11 aggregated vs 10 per-fill). The doc on applyAuctionAdjustmentsSQL states why the single trunc on the summed delta is kept: the skew is strictly below the end-of-window rate approximation already accepted on the same display-only fields.
The Down re-adds the narrowed CHECKs as NOT VALID, which is precisely the modifier that skips checking existing rows — so the stated precondition (no BLEND_* rows) was a comment, not a guarantee, and a rollback over Blend data left rows the restored constraints forbid: unwritable, and fatal to whole stateChanges pages in the pre-Blend resolver. A DO block now probes both the category and reason lists (bloom sparse indexes serve both) and raises before touching any constraint. All four DROP CONSTRAINTs gain IF EXISTS so both directions are re-runnable. The Up keeps NOT VALID: validating would scan every columnstore chunk under an AccessExclusiveLock.
…claim token Verified against blend-contracts-v2 @ ba22b487: fill_auction (pool/src/auctions/*.rs): fill_bad_debt_auction moves the bid dTokens from the backstop's Positions to the FILLER's — the filler assumes the debt — while the lot (backstop LP tokens) is drawn straight to the filler's wallet and never touches pool Positions. fill_interest_auction settles entirely outside pool Positions (bid donated to the backstop, lot paid from the reserves' backstop_credit, captured by the ResData entry snapshot). The decoder previously folded the user side of every asset for all auction types and mirrored to the filler only for type 0: a type-1 filler's net_borrowed missed the assumed debt, and type-2 fills fabricated lot adjustments against the backstop-address row (in the wrong units — underlying, valued as bTokens). Folds now mirror the on-chain Positions moves exactly: type 0 both sides, type 1 bid-only both sides, type 2 none. backstop claim (backstop/src/contract.rs -> emissions/claim.rs): the event's amount is execute_claim's return — the Comet LP tokens minted and auto-deposited (per-pool deposit events are emitted alongside) — never raw BLND. The row's token_id is now NULL with units backstop_lp in key_value, matching every other backstop-LP-denominated row, instead of mislabeling LP amounts as BLND.
Decode a ClaimFold from each pool/backstop claim event and accumulate it into the staged pool/backstop claimed-total maps during current-state indexing, persisting via PoolClaimed/BackstopClaimed.BatchApplyDeltas. History mode is unchanged — it still records the CLAIM feed rows but folds no totals.
… swap caveats Audit of the decoders/processor against blend-contracts-v2 @ ba22b487 found no behavioral gaps; this records the three residual findings: - test: a backstop claim op also emits one genuine deposit event per claimed pool (auto-restake) — 1 CLAIM + N BACKSTOP_DEPOSIT rows, a single account-wide claimed-total fold, no double count - godoc: TTL-evicted temporary Auction entries are invisible to ingestion (tx-meta-only reads), so an unfilled expired auction leaves a stale blend_auctions row - godoc: an emitter backstop swap requires updating the canonical backstop pin and migrating backstop-derived state
ToAPY guarded only the negative base; the positive side overflows math.Pow to +Inf once borrow APR exceeds ~2,186 at 365 daily periods, and ir_mod — an unclamped i128 read straight from permissionless reserve config — can push APR there. gqlgen cannot marshal Inf/NaN (the field errors and strict clients drop the whole response), and a zero USD side multiplied into an Inf APY turns netApy into NaN. ToAPY now reports non-finite results via an ok flag, computeReserveRates carries *float64 APYs (nil when unrepresentable), and both USD-weighted aggregates treat a nil APY on a priced reserve as uncomputable — netApy/ interestApy go null rather than silently weighting that reserve at 0% yield, the same convention a missing oracle price follows. Covered by boundary tests at the 2,186/2,187 threshold and resolver tests that seed a poisoned ir_mod and assert null APYs with no GraphQL error.
buildPoolPosition's contract is that totals go nil rather than silently understate, but its rp == nil branch — a position whose blend_reserves row doesn't exist — continued without clearing the known flags, shrinking suppliedUsd/usdValue while omitting the reserve from the response. Worst case a live position rendered as usdValue 0 with reserves: [], indistinguishable from a closed one. The case is reachable: a staging window carrying a new reserve's ResConfig without its ResData writes no row (BatchUpdateConfig is a plain UPDATE), so positions can precede their reserve. The branch now clears both flags, matching the missing-price rule; buildReservePosition's doc drops its 'shouldn't happen in practice' framing for the real mechanism, and a regression test pins the nil propagation (mutation-verified: restoring the bare continue fails it).
…ot zero An emissions-only write creates the pool's blend_backstop_pools row with zero balance columns (BatchUpsertEmissions inserts just the emis_* columns; the DDL defaults shares/tokens/q4w to '0'), and the balance half lands independently — so a user provably holding shares could render lpTokens "0" / usdValue 0, indistinguishable from a closed position, with a stale emission index silently applied to their real shares. The row's mere existence proved nothing. buildBackstopPosition now keys the shares→LP conversion on the rate being known (poolShares > 0; a user holding shares proves deposits exist, so zero pool shares is always a data gap): lpTokens and usdValue go null — lpTokens becomes nullable in the schema — and every Q4W entry follows the same rule; emissionsEarnedBlnd keeps the stored-index floor. The pool catalog's backstopUsdForPool applies the matching rule: zero balances beside a populated emission half read null (backstop emissions only crank for reward-zone pools, which requires deposits), a missing row stays a genuine $0, and zero tokens with nonzero shares (a backstop fully drawn for bad debt) stays a genuine $0. Regression tests cover the emissions-only row, the missing row, and the catalog split; the gate is mutation-verified.
…on fan-out One blendPositions resolution issues 13 DB queries (errgroup waves of 6+4+3, no dataloader by design) and one blendPool resolution 7, but both fields sat at the default 1+childComplexity — so an account-history page selecting stateChanges → account → blendPositions priced at ~500 while executing 13 queries per state-change row against a pgxpool of MaxConns=10. The fields now carry their query count as an additive flat cost (+13 / +7), which keeps the no-double-charge property for list depth while charging nested repetition its real fan-out — the amplified shape now prices at 1,701 (pinned by a new test case). The three blendPositions waves are also SetLimit-capped (4/4/3) so a single request can never drain the shared connection pool.
Blend pool/positions/earn-options query builders and client methods, plus client-side DTOs and inline fragments for the eight concrete Blend state-change types, registered through the generic unmarshal dispatch and the schema-validation test.
…or tables Read blend_pool_claimed/blend_backstop_claimed via the new GetByAccount readers after phase-2 live ingestion and assert the supplier's pool claim and whale's backstop claim folded positive totals. Guard that phase-1 (no claims) leaves both accumulator tables empty.
…s survive The canonical-backstop pin resolves to empty on the standalone network, which made the processor drop every backstop-shaped entry and event as a non-canonical impostor — BlendMigrationTestSuite failed on the whale's missing blend_backstop_positions row. The suite deploys the backstop from the master account (keypair.Root of the passphrase) with a fixed salt, so its address is a deterministic function of the passphrase alone. Pin it in canonicalBackstopAddress the same way blndTokenAddress pins the standalone BLND SAC: unit test + a deploy-time assertion in SetupBlendStack.
Absorbs the review-round API changes on blend/pr5-graphql: drops the GetBlendEarnOptions client method, query, and types (the query was removed - earn discovery composes from blendPools); BlendPool.status becomes the BlendPoolStatus enum string; BlendReservePosition's single emissionsApr splits into emissionsSupplyApr/emissionsBorrowApr. Claimable-emission assertions were already sign-based, so the projected-to-now claimable math needs no test changes.
executeSorobanOperationAs was a near-verbatim copy of executeSorobanOperation that fixed two real defects only in the copy: it preserved simulation-assigned auth nonces (nonce 0 is one-shot per address) and re-simulated after signing so MinResourceFee reflects the signed entries' size. One executeSorobanOperation(op, source, extraSigners, retries) now carries both behaviors for every caller. Sequence resolution branches on the source: the master account drives SharedContainers' locally tracked counter, any other actor's sequence is fetched from RPC. Since every master submission now advances the local counter, the As-path drift SyncMasterSequence existed to repair can no longer occur, so it and its re-sync call sites are removed. Auth entries an available keypair cannot sign now fail loudly naming the required address (the old shared path silently left them unsigned); source-account-credentialed entries pass through unchanged as before.
…mit for them The wbclient full-detail account-history query measures 10,101 at first:100 — over the previous 10,000 default — because the shared stateChangeFragments const gained 8 Blend fragments (+27 fields per state-change node, ×100 edges). The regression test could not catch this: it asserted a hand-copied mirror of the SDK query that had drifted (no Blend fragments, aliases the builder never emits), and the integration container overrode the limit to 30,000. pkg/wbclient now exports Queries(), the exact documents the client sends, and both server-side guards consume it: schema validation (which previously missed the three Blend queries) and the complexity regression test, which prices every SDK query at the largest accepted page size against the flag's own FlagDefault. Measured: BlendPools=26,150, full-detail history=10,101, state-change queries=8,401, blendPositions=7,583. The default limit rises to 30,000, documented as sized for the SDK's heaviest shipped queries with the DoS tradeoff stated (deployments not serving Blend should lower it). The integration container no longer overrides GRAPHQL_COMPLEXITY_LIMIT, so the suite proves the shipped default serves the SDK's full-selection queries end-to-end.
The wallet_db_query_duration_seconds histogram topped out at 0.38s (ExponentialBuckets(0.0001, 2.5, 10)), so every multi-second bulk COPY landed in +Inf and histogram_quantile panels clipped at ~0.38s, under-reporting exactly the queries being optimized. Widen to ExponentialBuckets(0.0001, 3, 12) — 0.1ms through ~17.7s.
…-ledger lookup The oldest-ledger lookup (SELECT ledger_number FROM transactions ORDER BY ledger_created_at ASC LIMIT 1) has two consumers — backfill gap detection's left bound (GetOldestLedger) and the hourly reconcile_oldest_cursor TimescaleDB job — and neither had an ordered path: the transactions migration dropped TimescaleDB's default partition-column index as consumer-less, an audit that counted WHERE-clause consumers and missed that this query consumes the index's ordering. Without it the planner pulls the first row from every chunk instead of running an ordered ChunkAppend that stops at the oldest (12-187s on an 87GB DB; the hourly job alone accounted for 45% of all DB disk-read time and flushed shared_buffers every run). Keep the default index (fresh databases get it from create_hypertable; already-migrated environments need a one-time manual `CREATE INDEX transactions_ledger_created_at_idx ON transactions (ledger_created_at DESC)`) and drop the to_id tie-break from both query sites: close times increase strictly, so rows sharing a ledger_created_at carry the same ledger_number and the tie-break could only force an incremental sort on top of the index's pathkeys. Covered by a new test that runs the reconciliation job via run_job against out-of-order chunks.
…the schedule interval Each hypertable's columnstore policy is auto-created with an identical schedule, so all five fire at the same instant and compress their just-closed chunks concurrently — an I/O storm that starves the persist stage (measured on the loadtest rig: persist p50 0.68s -> 3.0s for 18 of every 60 minutes). Converge each policy onto a fixed schedule anchored to the interval grid with a distinct per-table slot offset, so at most one policy comes due at a time. Jobs already on their slot are left untouched across restarts, preserving next_start and run history.
This was referenced Aug 13, 2026
aditya1702
force-pushed
the
blend/pr6-integration-tests
branch
2 times, most recently
from
August 17, 2026 19:04
c7561b5 to
996e7a2
Compare
An error occurred while trying to automatically change base from
blend/pr6-integration-tests
to
blend/pr5-graphql
August 17, 2026 20:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three independent fixes, one commit each — reviewable in any order. Review time: ~15 min. One manual-DDL warning below.
Found while profiling live ingestion: a metric that clipped exactly the queries being optimized, a missing index that made an hourly job 45% of all DB disk reads, and five compression policies firing at the same instant.
1. Query-duration buckets clipped at 0.38 s
wallet_db_query_duration_secondstopped out at 0.38 s: every multi-second bulk COPY landed in+Infand p99 panels flattened. Widened to 0.1 ms – ~17.7 s.2. Oldest-ledger lookup had no ordered path
SELECT ledger_number FROM transactions ORDER BY ledger_created_at ASC LIMIT 1(backfill gap detection + the hourlyreconcile_oldest_cursorjob) had no index to walk — an earlier audit dropped TimescaleDB's default partition-column index after counting only WHERE-clause consumers, missing that this query consumes the index's ordering. The planner pulled the first row from every chunk: 12–187 s on an 87 GB DB; the hourly job was 45% of all DB disk-read time.Fix: keep the default index; drop the
to_idtie-break from both query sites (close times increase strictly — the tie-break could only force an extra sort). Tested by running the reconciliation job against out-of-order chunks.Warning
Manual DDL on already-migrated databases (fresh DBs get the index from
create_hypertable). Applied on the loadtest DB; pending on dev/staging/prod:Note
2025-06-10.2-transactions.sql(already shipped) is edited in place, not superseded. Deliberate: wallet-backend is pre-release, and the baseline files stay the single readable statement of each table's final shape.3. Compression policies all fired at once
All five hypertables' columnstore policies were auto-created on identical schedules and compressed their just-closed chunks concurrently — an I/O storm that took persist p50 from 0.68 s to 3.0 s for 18 of every 60 minutes on the rig. Each policy is now anchored to a distinct slot offset on the schedule grid, so at most one fires at a time; jobs already on their slot are untouched across restarts.
Fourth of four PRs from the loadtest campaign (replaces #682). Independent — branches directly off
blend/pr6-integration-tests, shares no code with #679/#684/#685, mergeable anytime. Combined rig result: 18,458 tx/s, process p99 0.992 s; PRs not separately re-benchmarked.