This repository was archived by the owner on Aug 3, 2026. It is now read-only.
test: executable repro for torn-shutdown corruption, plus a loaded-index-growth guard - #179
Merged
Conversation
added 2 commits
August 1, 2026 17:30
…owns Two tests from the 2026-08-01 agencyzero store corruptions. loaded_index_growth (passing, a guard): build a store, close it, load it, and append until the primary index must grow past the size it was loaded at, then hold every row addressable across one more reload. The production stores died with their primary.wt.idx frozen at the loaded size while .wt.data grew, so growth-on-a-loaded-table stays pinned. torn_shutdown (ignored, the repro): kill a writer child mid-write five times, then load and scan. Fails today two ways, both observed in production: a phantom all-zero row comes back from the scan, or the load dies inside data_bucket page parsing. The engine writes pages in place with no atomicity across an operation's data-and-index steps, and reads everything back through rkyv::access_unchecked, so an abrupt death (crash, SIGKILL, quit without wait_for_ops) leaves bytes that load as garbage and blow up later as SIGBUS. Un-ignore when the engine gets crash-consistent writes or validated loads. Run it with: cargo test -- --ignored test_store_survives_torn_shutdowns
The passing control for the torn-shutdown repro: the same schema taken through forty sessions of a few appends each, every one drained and closed cleanly, then held to the exact id set on a final scan. It passing narrows the corruption to abrupt deaths: clean generational aging alone does not drift, so a store that dies carried a tear from a kill, a crash, or an undrained exit somewhere in its history.
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two tests distilled from the four store corruptions agencyzero took on 2026-08-01.
test_store_survives_torn_shutdowns(ignored, the repro). Kills a writer child mid-write five times, then loads and scans the store in-process. It fails today in the two ways production failed:id: "\0\0\0\0\0\0\0\0") — an index link published for data that never got written, the same debris agencyzero pulled out of its store;data_bucketpage parsing (parse_general_header).Root cause, as far as the evidence goes: pages are written in place with no atomicity across one operation's data-and-index steps (the
PersistenceTaskDrop comment says as much), and everything is read back throughrkyv::access_unchecked, so torn bytes don't fail at load — they load as garbage whose wild relative pointers SIGBUS in whatever walks them later. Any abrupt process end (crash, SIGKILL, or a consumer quitting withoutwait_for_ops) can tear the store, and each SIGBUS then tears it further: that cascade is what ate agencyzero's store four times in one day.Run it with:
cargo test -- --ignored test_store_survives_torn_shutdowns(fails in ~4s). Un-ignore it the day the engine gets crash-consistent writes (WAL / shadow paging / ordered fsync) or validated-and-refusing loads (bytecheck at the parse boundary + page checksums). Those are the fix directions; both are engine-level projects rather than spot patches, which is why this PR ships the repro rather than an attempt.test_primary_index_grows_on_a_loaded_table(passing, a guard). The dead production tables shared a signature:primary.wt.idxfrozen at exactly the size it was loaded with (65536 in one, 229376 in the other) while.wt.datakept growing. Plain build-close-load-append does grow the index correctly, so this pins that invariant with exact-id-set assertions across a further reload, in the schema shape that died (String uuid primary key, duplicated String secondary, ~1K payloads).Preserved forensic material (poisoned stores, before/after copies) lives at
~/code/worktable-idx-growth-evidence-20260801/with the analysis in~/code/HANDOFF-worktable-idx-growth.md.