db: squash migrations - #26
Merged
Merged
Conversation
Fresh databases deadlocked during startup migrations: the old history
created continuous aggregates whose refresh/retention policy jobs start
running immediately (003/005/016), then dropped those aggregates in
later migrations (006/015/020). DROP MATERIALIZED VIEW ... CASCADE must
delete the policy jobs and races the TimescaleDB job scheduler —
relation locks vs per-job advisory locks, in opposite order. Which side
loses is a coin flip, so CI failed while production "worked" (there the
scheduler was the deadlock victim and simply restarted).
Replace the history with a baseline that creates only the final schema:
001_core extensions, ingested_raw_events (+events view,
hot columns, 1h retention), nodes, event_types
002_count_tables 14 per-group count tables (all 115 event types)
003_count_aggregates _1m/_1h continuous aggregates over count tables
004_union_views all_event_stats_30s/1m/1h, all_core_stats_1m
005_node_stats_services node_stats + event_services + 1m aggregates
006_trackers convergence, wp_tracking, DA stats/latency hists
007_onchain on-chain core/service/validator stats
No aggregate is ever dropped during migration, so the deadlock is
structurally impossible; the retry workaround in store.rs is reverted.
Verified schema-equivalent to the old history: both sets applied to
scratch databases, then diffed over normalized catalog state (columns
incl. ordinal positions, constraints, indexes, view definitions,
hypertable dimensions, compression settings, policy jobs, continuous-
aggregate materialized_only flags, seed rows) — empty diff. Full test
suite passes against the new baseline.
Existing databases must be recreated once: the _sqlx_migrations ledger
no longer matches (use clean-and-reset-all, or drop tart_test before
run-tests.sh). CI is unaffected — it always starts from fresh
containers.
The migration squash preserved a historical quirk: the aggregates for the five newest count groups (status, connection, block, ticket_low, wp_pipeline) ran with the default materialized_only = true, because the old migration that enabled real-time mode (017) predated their creation (020) and they were never revisited. As a result the all_event_stats_1m and _1h UNION views mixed freshness: 016-era branches served data up to the current minute while these five groups lagged by the refresh interval (up to ~2 min on the 1m tier, ~1 h on the 1h tier). Set materialized_only = false on all 28 count-table aggregates in the baseline, so every branch of the UNION views has the same freshness (node_stats_1m and service_stats_1m already had it). Edited in-place rather than as a new migration — the squash requires recreating every database anyway, so there is no ledger to preserve. The performance caveat from the old migration 017 carries over: the tail scan reads the raw count table for the un-materialized window on every query. If aggregate queries slow down on a 1024-node network, this setting is the first thing to check (per-view revert recipe in the migration comment).
s0me0ne-unkn0wn
approved these changes
Aug 5, 2026
s0me0ne-unkn0wn
left a comment
s0me0ne-unkn0wn
left a comment
There was a problem hiding this comment.
Looks clean, and the reasoning makes sense. I confess I didn't thoroughly read through all the SQL, but structurally everything seems to be in its place.
Comment on lines
+40
to
+42
| # NOTE: if your tart_test predates the migration squash (2026-08), drop it once | ||
| # first (`cargo sqlx database drop -y && cargo sqlx database create`) — the old | ||
| # _sqlx_migrations ledger doesn't match the squashed migration set. |
There was a problem hiding this comment.
Given that nobody writes tests by hand nowadays, this is better placed somewhere like AGENTS.md or CLAUDE.md. I wouldn't count on an agent developing something and adding a test for it to watch in this file for guidelines.
Contributor
Author
There was a problem hiding this comment.
I used that note in my tests recently :), yeah feel free to add agents or claude.md in some future work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
store.rsretry reverted).materialized_only=falseon all count aggregates (5 groups accidentally lagged their refresh interval), dead compression config dropped fromingested_raw_events.