perf(data): COPY account-link rows in primary-key order - #697
Closed
aditya1702 wants to merge 1 commit into
Closed
Conversation
The link rows are built by iterating nested maps, so the COPY hit the (account_id, ...)-led primary key in randomized order — one random btree descent per row, and on a cold cache a random page read each. Sorting the batch by (account_id, id) walks the index left-to-right instead, the same ordering the balance upserts already apply. Measured on the loadtest rig, the two link-table COPYs were the slowest persist siblings at ~400ms median per commit — 3x the parent transactions COPY at a third of its per-row work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Duplicate of #696, which survived the branch rename. |
Contributor
Author
|
Supersedes #696 (closed permanently by the head-branch rename to live-ingest-persist-opts). |
Contributor
Author
|
Holding off — this branch will get one PR together with the persist-phase optimization batch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(Recreates #696, closed by a head-branch rename.)
What
Sort the transactions_accounts / operations_accounts COPY batches by (account_id, id) before shipping them.
Why
The rows were built from nested map iteration, so each COPY paid a randomized btree descent per row on the account_id-led primary key. The balance upserts already sort by PK for the same reason.
Testing
go test ./internal/data/green; COPY order is correctness-neutral.🤖 Generated with Claude Code