GH-980: projection StopAndDrainAsync honours its timeout and skips queued pages

[jeremydmiller]

Closes #980. Follow-up to #953, which did this for subscriptions.

What changed

ProjectionExecution and GroupedProjectionExecution now drain the way SubscriptionExecutionBase does after #953:

• A drain lets the page in flight finish and skips the queued ones. A _draining flag is checked where a page starts: in processRangeAsync for both executions, and also in the grouping stage for GroupedProjectionExecution. A page still being grouped when the drain starts is dropped before it's applied. The next owner of the shard picks the skipped pages up from the stored progression.
• The wait is bounded by the token (DaemonSettings.StopAndDrainTimeout), and the execution is cancelled in a finally, so a timed-out drain cancels the page in flight. Its batch, including the progression update, doesn't commit.
• A page that finishes after cancellation doesn't call MarkSuccessAsync. In the grouped execution, ApplyPendingCacheUpdates still runs before that check, because the batch did commit.
• EnqueueAsync ignores pages posted after the drain started.

Paths checked

• Replays and composite stages (CompositeReplayExecutor, ExecutionStage) call ProcessRangeAsync directly. Composite member executions are never drained, so they're unaffected. A replay on a composite execution that is being drained stops at the next page, which is what the existing cancellation check did already once the drain cancelled.
• ProcessImmediatelyAsync (catch-up) skips pages once a drain has started, as subscriptions do since fix: subscription StopAndDrainAsync honours its timeout and skips queued ranges #953.
• CompositeProjectionSource's StopAndDrainAsync was already a no-op.
• Deferred rebuild writes (Deferred single-flush rebuild writes: one store op per aggregate per flush window (rebuild-optimization Phase 4) #525): skipped pages never reach the flush, and DisposeAsync / HardStopAsync still discard the accumulator. Committed progress only advances at a flush, so the next owner resumes from the last one. AggregationRunner.FlushDeferredRebuildWritesAsync is unchanged: a flush that commits just before cancellation still marks success in memory.

Tests

ProjectionExecutionStopAndDrainTests runs against the real ProjectionExecution and GroupedProjectionExecution, with gated substitute batches. Each case runs once per execution:

• the page in flight finishes and the queued pages are skipped;
• a timed-out drain cancels the page in flight and marks no progression;
• a page that finishes after its drain timed out marks no progression;
• a page enqueued after the drain started is ignored;
• ProcessRangeAsync, called directly, still applies the page and marks progression.

Negative control: with the source changes reverted, 6 of the 10 cases fail. The enqueue-after-drain and direct-range cases pass either way, as expected.

Local run

./build.sh test-events:

• net10.0: 1379 / 1379 pass.
• net9.0: 1378 / 1379 pass. The one failure is BlueGreenSideEffectGateTests.fresh_deploy_suppresses_side_effects_to_the_prior_version_mark_then_enables_them. That test uses its own RecordingExecution, not the code changed here, and it failed in 4 of 6 standalone reruns. It's a pre-existing race between the recorded page and the progress assertion.

Downstream

As with #953, Marten, Polecat and Fisher need a pin bump and a release note: JasperFx/marten#5610, JasperFx/polecat#744 and JasperFx/fisher#434. The release note in #981 currently says async projections are unchanged; it needs updating if this ships in the same release.

🤖 Generated with Claude Code

https://claude.ai/code/session_01J8KxC18Q51z5w4u6YARwe4

[jeremydmiller]

Added a test-only commit: the BlueGreen gate harness announced each page before recording its progress, so fresh_deploy_suppresses_side_effects_to_the_prior_version_mark_then_enables_them read the old mark under full-suite load (failed 3/3 on this branch, 0/3 on main; 3/3 green after the fix). Local gate (no GitHub Actions today): full ./build.sh --root green, all 15 targets.

🤖 Generated with Claude Code