Persist progression when a composite single-pass replay ends on an empty page

[erdtsieck]

Problem

We hit this in production on JasperFx.Events 2.81.0 + Marten 9.47.0 with per-tenant partitioned events: 269 tenant agents of one composite projection ended up Stopped with ProgressionProgressOutOfOrderException, and nothing restarted them.

The composite has a composite-level IncludeType filter. When its agent starts continuous with no progression row (LastCommitted == 0) and without GateSideEffectsBehindPriorVersion, SubscriptionAgent takes the single-pass replay path (SubscriptionAgent.cs, the CommandType.Start branch that calls TryBuildReplayExecutor) through CompositeReplayExecutor.

For a tenant whose events are all filtered out, the first replay page is empty. CompositeReplayExecutor then only calls controller.MarkSuccessAsync(ceiling). That moves LastCommitted to the ceiling in memory, but no progression row is written. The next continuous page has floor = ceiling, so Marten's ProjectionBatch.RecordProgress takes the update path (floor != 0). UpdateProjectionProgress (... where name = ? and last_seq_id = ?) matches no row and throws ProgressionProgressOutOfOrderException, and the agent stops.

The same thing happens when the last replay page is exactly BatchSize and the next page is empty: the row is left at the last event's sequence while memory is at the ceiling.

Root cause

src/JasperFx.Events/Projections/Composite/CompositeReplayExecutor.cs: the page.Count == 0 branch advanced progression with MarkSuccessAsync only. Every other page goes through _execution.ProcessRangeAsync, which starts a batch (where the store records the composite's progress: insert from floor 0, update otherwise), records each member's progress in ExecutionStage, executes, and only then calls MarkSuccessAsync. The empty page skipped all of that, so the stored and in-memory positions diverged.

Fix

For the empty page, commit an empty EventRange(page.Floor, ceiling) through the composite execution and stop, instead of only calling MarkSuccessAsync. Floor 0 now inserts the composite and member rows, and a later floor updates the row from where the previous page left it. Termination is unchanged: the pass still ends at the ceiling on the first empty page.

The ceiling <= floor branch at the top stays as it is; with nothing in the store the next continuous page still starts at floor 0 and inserts.

Tests

• New CompositeReplayExecutorProgressionTests drives a real CompositeExecution with a small fake progression table that behaves like Marten's (insert from floor 0, otherwise update only where the row is at the floor, else ProgressionProgressOutOfOrderException):

  • a replay over only filtered-out events leaves the composite and member rows at the ceiling, and the following continuous page commits cleanly;
  • a replay that ends on a full page followed by an empty page leaves the row at the ceiling, not at the last event.

  Both fail on main (row missing / row at 2 instead of 10) and pass with the fix.

• CompositeReplayExecutorTests.advances_to_ceiling_when_no_events_match_below_high_water pinned the old behaviour (no ProcessRangeAsync); it now asserts the empty range (0, 25] goes through the execution, renamed accordingly.

• EventTests: 1384/1384 passing on net9.0 and net10.0.

I did not add a Marten-level test; the progression semantics are Marten's, and I modelled them in the fake rather than pulling a store into JasperFx.

Downstream we have mitigated this for now by turning on GateSideEffectsBehindPriorVersion, which keeps that composite off the single-pass replay path.

https://claude.ai/code/session_01C5ySTvFt8fb8TmQakcKRx9