FlatKV: background read/fold for account data - #4155
Conversation
cody-littley
changed the title
PR SummaryHigh Risk Overview FlatKV stops batch-reading accounts during Reviewed by Cursor Bugbot for commit d38fe4d. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4155 +/- ##
==========================================
- Coverage 66.60% 65.80% -0.80%
==========================================
Files 2196 2103 -93
Lines 169188 160603 -8585
==========================================
- Hits 112692 105690 -7002
+ Misses 56355 54772 -1583
Partials 141 141
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d38fe4d. Configure here.
| shard.FoldStagedValues(folds, updater, version) | ||
| }) | ||
| } | ||
| return nil |
There was a problem hiding this comment.
Close races in-flight account folds
Medium Severity
BatchUpdate returns after staging and submits folds on miscPool with no wait, and Close was not updated to drain versionLatches for the current unsealed version. A legal BatchUpdate then Close sequence can let a fold read the backing DB after Close has shut it down, which breaks the guarantee that no manager work touches that DB once Close returns.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d38fe4d. Configure here.
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
There was a problem hiding this comment.
Moving the account read/fold off the execution thread is well structured — staging under the shard lock, pending-value handles checked on every read path, and per-version latches that gate the diff consumers — and the new tests (including the differential opUpdate op) cover the staging/chaining/delete/failure semantics well. The blocking gap is lifecycle: the folds are fire-and-forget work that nothing drains, so Close can tear down the read pool and the database underneath an in-flight fold.
Findings: 1 blocking | 2 non-blocking | 2 posted inline
Blockers
- None at the file/PR level.
- 1 blocking issue(s) flagged inline on specific lines.
Non-blocking
- [suggestion]
ApplyChangeSetsno longer detects a corrupt stored account row —newAccountUpdateronly parses the changeset, and deserialization of the row being folded onto now happens on a fold worker. The block is therefore accepted and the failure resurfaces later as a bricked manager (and a panic out ofCommitStore.Get, asTestCrashRecoveryCorruptedAccountValueInDBnow records) rather than as an error attributable to the apply. The node still halts before publishing a hash, so this is a change in failure surface rather than a correctness hole, but it is worth calling out explicitly in theCommitStore.ApplyChangeSetscontract so operators reading a fold-failure brick know which block produced it. - 1 suggestion(s)/nit(s) flagged inline on specific lines.
| continue | ||
| } | ||
| shard := c.shards[shardIndex] | ||
| c.miscPool.Submit(func() { |
There was a problem hiding this comment.
[blocker] These fold tasks are the first manager-owned background work that touches the database and that nothing waits for, and shutdown has no drain for them.
closeInternal waits only for the lifecycle runner, then cancels, takes the shards out of service and calls c.db.Close(). The lifecycle runner does no final flush on its exit path, so folds staged at the current (unsealed) version are never awaited by anything — GetDiffsForVersions's latch wait only covers versions that actually get flushed or hashed. CommitStore.Close then closes readPool before miscPool (store_lifecycle.go), and its own comment states the invariant this breaks: "Store Close does not return until no store-owned goroutine will touch the database again."
Concrete sequence, with a caller that has fully quiesced (last ApplyChangeSets returned, no Commit yet) and then calls Close:
- A fold worker gets past
schedulePriorValueReadsUnlocked(out-of-service check passes) with a non-emptypendingReadlist. closeInternalruns to completion: cancel,TakeOutOfService,c.db.Close().s.readPool.Close()runs (fixed pool,closed = true).- The worker reaches
readCache.ResolveBatchUnlocked, which callsc.readPool.Submit(...)unconditionally —fixedPool.Submitpanics with"threading: submit on closed pool".
A slightly wider variant of the same window: the worker is already parked in InterruptiblePull while its scheduled read task calls readFromDB against the handle step 2 just closed.
The fix belongs at the choke point rather than at each caller: track outstanding folds on the manager (a WaitGroup incremented at stage time, decremented in recordFoldsUnlocked/FailStagedFolds) and drain it in closeInternal before c.db.Close(), so ViewManager.Close again returns only once no manager-owned goroutine will touch the database or either pool. A test that calls BatchUpdate and then Close without an intervening Commit/flush would pin it.
| // | ||
| // Must be called without the shard lock held: it acquires versionLock, and the established order is | ||
| // versionLock before any shard lock. | ||
| // reportFoldFailure handles a fold that could not produce its value by bricking the manager. Separate |
There was a problem hiding this comment.
[suggestion] reportFoldFailure was inserted between reportReadFailure's doc comment and its declaration. The result is that reportReadFailure now has no godoc at all, and its three paragraphs — including the load-bearing "Must be called without the shard lock held: it acquires versionLock, and the established order is versionLock before any shard lock" — read as documentation for reportFoldFailure.
That invariant does apply to both functions (it is why shard.FailStagedFolds releases the lock before calling reportFoldFailure), so it should be stated on each rather than silently transplanted. Move the new function below reportReadFailure, leaving the existing comment attached to its own declaration.
1 participant
Describe your changes and provide context
Move account read/fold workflow off of the execution thread.