Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 22 additions & 6 deletions src/content/docs/Immediate.Jobs/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ sealed class JobAttribute : Attribute

enum OverlapPolicy { Skip, Queue, Concurrent }
enum BackoffStrategy { Fixed, Exponential, ExponentialJitter }
enum JobState
{
AwaitingContinuation, AwaitingParameters, Scheduled, Pending, Active,
Succeeded, Failed, Cancelled, Skipped
}

sealed class QueueDefinitionAttribute : Attribute
{
Expand Down Expand Up @@ -128,8 +133,9 @@ ValueTask<JobHandle> AddToBatchAsync(
`ContinuationTrigger.Success` waits for every parent to succeed. `Failure` waits for every parent
to become terminal and then runs when at least one failed. `Complete` waits for every parent to
become terminal regardless of outcome. A `BatchHandle` is a single parent whose state becomes
`Failed` when any item in the completed batch failed; cancellation alone does not satisfy
`Failure`.
`Failed` when any item in the completed batch failed; cancellation or skipped branches alone do
not satisfy `Failure`. When a `Success` or `Failure` condition cannot be satisfied, the child and
any ineligible descendants become terminal `Skipped` records.

| `ContinuationOptions` | Batch membership | Effect on the current job's existing continuations |
| ------------------------------- | ---------------- | ---------------------------------------------------------------- |
Expand Down Expand Up @@ -220,11 +226,19 @@ interface IJobBatchMonitor
string batchId, BatchMemberQuery query, CancellationToken token = default);
ValueTask<BatchGraph?> GetGraphAsync(string batchId, CancellationToken token = default);
}

sealed record BatchStatus(
string Id, BatchState State,
int Total, int Succeeded, int Failed, int Cancelled, int Skipped, int Remaining,
DateTimeOffset CreatedAt, DateTimeOffset? StartedAt, DateTimeOffset? CompletedAt,
double FractionSettled
);
```

`BatchMemberQuery` and `JobBatchQuery` contain optional state, `Skip`, and `Take = 100`.
`JobStatus`, `BatchStatus`, `BatchMemberStatus`, `BatchGraph`, `BatchGraphNode` and
`BatchGraphEdge` are immutable monitoring records.
`BatchGraphEdge` are immutable monitoring records. `FractionSettled` includes every terminal
outcome, including `Skipped`.

## Dashboard

Expand Down Expand Up @@ -281,18 +295,20 @@ each `ScheduledJobCapture<T>` contains `Id`, `Payload`, `RunAt`, and `GroupId`.
it exposes `Services`, `Storage`, `TimeProvider`, and `Batches`. Operations are `DrainAsync`, both
`AdvanceTimeAndDrainAsync` overloads, `QueryJobsAsync`, both `GetJobAsync` overloads, both
`AssertEnqueuedAsync<T>` overloads, `AssertBatchCommittedAtomicallyAsync`,
`AssertContinuationReleasedAfterAsync`, `AssertCascadeCancelledAsync`, and
`RunThroughPipelineAsync<T>`.
`AssertContinuationReleasedAfterAsync`, `AssertCascadeSkippedAsync`, its compatibility alias
`AssertCascadeCancelledAsync`, and `RunThroughPipelineAsync<T>`.

## Custom storage contracts

| Interface | Atomic responsibilities |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `IJobStorage` | Initialize; enqueue; lease/acquire/renew; persist execution telemetry; complete/fail; query/status; retry/delete/purge; heartbeat and health. |
| `IRecurringJobStorage` | Upsert/remove/pause/resume schedules; identify due rows; uniquely materialize each occurrence; reconcile obsolete code-defined schedules. |
| `IJobGraphStorage` | Atomically enqueue batches/edges; settle and release/cascade dependencies; add mid-run members; monitor/cancel/delete/purge graphs. |
| `IJobGraphStorage` | Atomically enqueue batches/edges; settle and release/skip dependencies; add mid-run members; monitor/cancel/delete/purge graphs. |
| `IJobStorageReplica` | Restore durable records and mirror explicit acquisitions for the single-server wrapper. |

`StorageCapabilities` flags are `Queue`, `Recurring` and `Graph`; call
`storage.GetCapabilities()` to detect the optional interfaces. Low-level `JobRecord`, acquisition,
definition and graph persistence records are provider contracts, not application scheduling APIs.
`IJobStorage.RetryAsync` accepts `Failed` and `Scheduled`: a scheduled invocation is moved to
`Pending` immediately while retaining its attempt count and latest failure details.
21 changes: 13 additions & 8 deletions src/content/docs/Immediate.Jobs/batches-and-continuations.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,15 +102,18 @@ handles from unrelated open batches are rejected. `ScheduleAfterAsync(BatchHandl
the entire prior batch. `batches.Begin(previousBatch, trigger)` creates a follow-up batch whose
root members all depend on it.

| `ContinuationTrigger` | Child is released when… |
| --------------------- | ---------------------------------------------------------------------------- |
| `Success` | all parents succeed; otherwise the child is cascade-cancelled. |
| `Failure` | all parents are terminal and at least one failed; otherwise it is cancelled. |
| `Complete` | all parents reach any terminal state. |
| `ContinuationTrigger` | Condition and unmatched outcome |
| --------------------- | ------------------------------------------------------------------------------------ |
| `Success` | Released when every parent succeeds; skipped when a parent settles unsuccessfully. |
| `Failure` | Released after every parent settles if at least one failed; otherwise skipped. |
| `Complete` | Released after every parent reaches any terminal state; it has no unmatched outcome. |

For a continuation attached to a `BatchHandle`, the batch is one parent. It becomes `Failed` after
all of its items finish when **any item failed**, so a `Failure` continuation is released in that
case. Cancelled items alone make the batch `Cancelled` and do not satisfy `Failure`.
case. Explicitly cancelled items alone make the batch `Cancelled`, while skipped branches do not
fail the batch. Neither outcome satisfies `Failure`. Conditional branches that are not selected
become terminal `Skipped` records, and skipping propagates through descendants whose own triggers
cannot be satisfied.

## Expanding a running workflow

Expand Down Expand Up @@ -159,5 +162,7 @@ except for detached scheduling, the current job must belong to a batch. `IJOB001

</Callout>

Monitor a graph through `IJobBatchMonitor.GetStatusAsync`, `QueryMembersAsync` and `GetGraphAsync`,
or use the dashboard's workflow view and batch cancel/delete operations.
Monitor a graph through `IJobBatchMonitor.GetStatusAsync`, `QueryMembersAsync` and `GetGraphAsync`.
`BatchStatus` counts succeeded, failed, cancelled and skipped members separately; a batch can
succeed when every executed member succeeded even if conditional branches were skipped. The
dashboard exposes the same progress and workflow states alongside batch cancel/delete operations.
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ support graph workflows or fair queues.

Storage initialization is idempotent provider startup, not a substitute for controlled production
schema evolution. Keep migrations/bootstrap and provider packages at the same preview revision as
the core package.
the core package. The batch table now includes a non-null `SkippedCount` column; existing SQL
schemas created before that field need an application-owned migration that defaults it to zero.
Comment thread
dukesteen marked this conversation as resolved.

</Callout>
9 changes: 6 additions & 3 deletions src/content/docs/Immediate.Jobs/creating-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,12 @@ rejected. Interfaces, abstract types, open generics, pointers, delegates, ref-li
unsupported `System` types are also rejected.

Prefer records or classes with a public constructor whose parameters match their public members.
Constructor availability is not currently an analyzer error, so a type that receives metadata but
cannot be constructed by `System.Text.Json` can still fail when a worker deserializes it. Generated
metadata is used for both serialization and Native AOT; no reflection fallback is needed.
Public `init` properties that are not constructor parameters are also supported, including
optional and `required` properties; generated metadata initializes them through an object
initializer. Constructor availability is not currently an analyzer error, so a type that receives
metadata but cannot be constructed by `System.Text.Json` can still fail when a worker deserializes
it. Generated metadata is used for both serialization and Native AOT; no reflection fallback is
needed.

A payloadless job receives `EmptyJobRequest`:

Expand Down
19 changes: 14 additions & 5 deletions src/content/docs/Immediate.Jobs/dashboard-and-monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,18 @@ Without `RequireAuthorization`, every dashboard endpoint is development-only and
other environments. Treat the dashboard as an administrative surface: it exposes payloads,
errors, identifiers and mutations. A named policy applies to UI assets and APIs together.

The UI shows queue/state totals, recent history, jobs and details, recurring schedules, scheduler
servers, batches and workflow graphs. Graph views appear only for graph-capable storage.
The UI shows queue/state totals, including skipped work, recent history, jobs and details,
recurring schedules, scheduler servers, batches and workflow graphs. Graph views appear only for
graph-capable storage.

## Dashboard UI

### Inspect jobs

The Jobs view lists recent executions and their current state. Select a job to inspect its
invocation, execution metadata, payload, errors and any application-defined telemetry links.
Failed jobs offer **Retry**; scheduled first attempts and delayed retries offer **Run now**, which
moves the existing invocation to `Pending` without changing its attempt or failure history.

<figure class="not-prose my-8">
<img
Expand Down Expand Up @@ -80,8 +83,9 @@ invocation, execution metadata, payload, errors and any application-defined tele

### Follow batch workflows

The Batches view visualizes the jobs in a batch and the continuations between them. Select a node
to inspect that job without losing the surrounding workflow context.
The Batches view visualizes the jobs in a batch and the continuations between them. Progress and
workflow nodes distinguish skipped conditional branches from explicitly cancelled work. Select a
node to inspect that job without losing the surrounding workflow context.

<figure class="not-prose my-8">
<img
Expand Down Expand Up @@ -139,7 +143,7 @@ All paths below are relative to the mapped prefix.
| `GET /api/jobs` | Filter by `state`, `queue`, `search`; `skip`; `take` 1–200 (default 50). |
| `GET /api/jobs/{jobId}` | Latest durable record. |
| `GET /api/jobs/{jobId}/telemetry-links` | Configured trace/log destinations. |
| `POST /api/jobs/{jobId}/retry` | Retry failed work. |
| `POST /api/jobs/{jobId}/retry` | Retry failed work or run scheduled work now. |
| `DELETE /api/jobs/{jobId}` | Delete terminal work. |
| `GET /api/recurring` | Recurring schedules. |
| `POST /api/recurring/{name}/trigger` | Materialize an immediate invocation. |
Expand All @@ -154,6 +158,11 @@ All paths below are relative to the mapped prefix.
| `GET /api/events` | SSE `state` snapshots at `UpdateInterval`. |
| `GET /api/batches/{id}/stream` | SSE `status` and `graph` events on change. |

Successful retry, delete, pause, resume and batch mutations return `204`; recurring trigger returns
`202`. A mutation returns `404` when its job, batch or recurring schedule does not exist (or the
provider lacks the required capability), and `409` when the resource exists but its lifecycle
state does not allow the operation.

SSE sends `retry: 3000`, disables proxy buffering and ends when the request is aborted. It is a
poll-backed live view, not a durable event log; clients must refresh after reconnecting.

Expand Down
34 changes: 19 additions & 15 deletions src/content/docs/Immediate.Jobs/delivery-guarantees.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,17 @@ scheduler, accepting that the relay itself also needs idempotency.

## Lifecycle

| State | Meaning |
| ---------------------- | ----------------------------------------------------------- |
| `Pending` | Due for acquisition now. |
| `Scheduled` | Persisted for a future `DueAt`. |
| `AwaitingContinuation` | Waiting for parent jobs or a parent batch. |
| `Processing` | Acquired by a worker under a renewable lease. |
| `Succeeded` | The attempt and completion transition succeeded. |
| `Failed` | Attempts exhausted or an operator-visible terminal failure. |
| `Cancelled` | Cascade- or batch-cancelled terminal work. |
| State | Meaning |
| ---------------------- | ----------------------------------------------------------------------- |
| `AwaitingContinuation` | Waiting for parent jobs or a parent batch. |
| `AwaitingParameters` | Reserved for future deferred-input work; currently never produced. |
| `Scheduled` | Persisted for a future `DueAt`. |
| `Pending` | Due for acquisition now. |
| `Active` | Acquired by a worker under a renewable lease. |
| `Succeeded` | The attempt and completion transition succeeded. |
| `Failed` | Attempts exhausted or an operator-visible terminal failure. |
| `Cancelled` | Terminal work stopped by an explicit cancellation, including a batch. |
| `Skipped` | Terminal work not selected by a continuation or recurring overlap rule. |

An acquisition increments `Attempt`. On failure, the worker calls `FailAsync` with either the next
retry time or no retry after `MaxAttempts`. Fixed and exponential backoff use `BackoffBase`;
Expand All @@ -48,12 +50,14 @@ expiry and recovery.

## History and operations

Defaults are 24 hours for succeeded jobs/batches, seven days for failed or cancelled jobs/batches,
and one hour between purge passes. Set `SucceededRetention`, `FailedRetention`,
Defaults are 24 hours for succeeded jobs and batches; seven days for failed, cancelled or skipped
jobs and for failed or cancelled batches; and one hour between purge passes. Set
`SucceededRetention`, `FailedRetention`,
`BatchSucceededRetention`, `BatchFailedRetention` and `PurgeInterval` on
`ImmediateJobsOptions`. Zero retention is valid; negative retention is rejected.

Operators can retry a failed job, delete a terminal job, cancel a non-terminal batch and delete a
terminal batch through provider operations or the dashboard. Active/non-terminal records reject
unsafe delete/retry mutations with a conflict. Batch retention removes its members and edges as a
unit.
Operators can retry a failed job, move a scheduled invocation to `Pending` immediately, delete a
terminal job, cancel a non-terminal batch and delete a terminal batch through provider operations
or the dashboard. Fast-forwarding a scheduled retry preserves its attempt count and latest failure
details. Other non-terminal records reject unsafe delete/retry mutations with a conflict. Batch
retention removes its members and edges as a unit.
Comment thread
dukesteen marked this conversation as resolved.
1 change: 1 addition & 0 deletions src/content/docs/Immediate.Jobs/diagnostics.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ Some facts depend on runtime values or durable state and cannot be diagnosed at
detects multiple-process replica drift;
- unknown stored job names fail terminally because no generated definition can execute them;
- unknown context slices are logged and skipped so rolling deployments can continue;
- dashboard mutations return HTTP 404 for an unknown job, batch or recurring schedule;
- retry/delete/cancel operations reject incompatible lifecycle states with
`ImmediateJobException` (HTTP 409 in the dashboard).

Expand Down
8 changes: 5 additions & 3 deletions src/content/docs/Immediate.Jobs/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ generators.

The hosted service initializes storage and recurring definitions, then builds acquisition requests
from queue priority plus node, queue and job capacity. Storage atomically changes eligible due work
to `Processing`, assigns a worker/lease and increments attempts. Distributed providers coordinate
to `Active`, assigns a worker/lease and increments attempts. Distributed providers coordinate
this in the shared backend; single-server mode acquires from memory and mirrors ownership to its
durable replica.

Expand All @@ -50,14 +50,16 @@ generated handler. The call enters Immediate.Handlers behaviors and ends at the

Success atomically records completion and any buffered mid-execution continuations. Failure stores
the exception and either schedules a retry or leaves a terminal failure. Settling a graph node
releases eligible children or cascade-cancels children whose trigger can no longer be satisfied.
releases eligible children or marks unselected branches `Skipped`. Release and recursive skipping
are committed in the same transaction as the parent's terminal transition.

## Recurring materialization

Code-defined schedules are reconciled at startup. The recurring loop asks storage for due
schedules; the provider atomically materializes a uniquely keyed occurrence and advances its next
run. This keeps multiple distributed nodes from creating the same occurrence. Overlap policy is
evaluated against active occurrences of the same schedule.
evaluated against active occurrences of the same schedule; `Skip` persists a terminal skipped
occurrence so monitoring retains the scheduling decision.

## Generated JSON, trimming and Native AOT

Expand Down
10 changes: 5 additions & 5 deletions src/content/docs/Immediate.Jobs/recurring-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,11 @@ dashboard can trigger, pause and resume existing schedules.

## Overlap policy

| Policy | When the previous occurrence is still active |
| ------------ | ----------------------------------------------------------------------- |
| `Skip` | Do not materialize this occurrence. |
| `Queue` | Materialize it and let it wait. |
| `Concurrent` | Allow both invocations to execute, subject to other concurrency limits. |
| Policy | When the previous occurrence is still active |
| ------------ | -------------------------------------------------------------------------- |
| `Skip` | Persist the occurrence as terminal `Skipped` history without executing it. |
| `Queue` | Materialize it and let it wait. |
| `Concurrent` | Allow both invocations to execute, subject to other concurrency limits. |

Materialization is coordinated in durable storage, so `Recurring` capability is required. Redis
and the SQL providers support it; graph support is unrelated.
Expand Down
3 changes: 2 additions & 1 deletion src/content/docs/Immediate.Jobs/testing-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ Assert.Equal(JobState.Succeeded, (await harness.GetJobAsync(handle, cancellation
exposes `Storage`, `TimeProvider`, `Services` and `Batches`.

For graphs, use `AssertBatchCommittedAtomicallyAsync`,
`AssertContinuationReleasedAfterAsync` and `AssertCascadeCancelledAsync`. Call
`AssertContinuationReleasedAfterAsync` and `AssertCascadeSkippedAsync`.
`AssertCascadeCancelledAsync` remains as a compatibility alias. Call
`RunThroughPipelineAsync<TPayload>` when a test already has a record/payload and needs to execute
its generated invoker through the real DI/behavior pipeline.

Expand Down