-
Notifications
You must be signed in to change notification settings - Fork 9
Document the built-in scheduler component plugin (5.2) #592
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jcohen-hdb
wants to merge
5
commits into
main
Choose a base branch
from
scheduler-component-docs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+138
−0
Open
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
4ba3063
Document the built-in scheduler component plugin
jcohen-hdb 0ded987
Use snapshot and external-API examples instead of record cleanup
jcohen-hdb 0279a40
Add YAML quoting guidance for cron expressions and handler references
jcohen-hdb af1a837
Align scheduler docs with the implementation contract (review findings)
jcohen-hdb caf0396
Spread untrusted API payload before explicit keys in scheduler example
jcohen-hdb File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,124 @@ | ||
| --- | ||
| title: Scheduler | ||
| --- | ||
|
|
||
| # Scheduler | ||
|
|
||
| <VersionBadge version="v5.2.0" /> | ||
|
|
||
| The scheduler is a built-in plugin that runs recurring jobs declared in a component's configuration. Harper invokes a designated export from your component on a cron or interval schedule. In a cluster, execution is leader-coordinated: under normal operation each occurrence runs once, on a single automatically elected node, rather than once per node or worker. During leadership failover, daylight-saving fall-back, or split-brain recovery an occurrence can occasionally be delivered more than once - which is why handlers must be idempotent (see below). | ||
|
|
||
| Use it for the recurring work applications otherwise hand-roll with `setInterval`: taking daily snapshots of a dataset, pulling from an external API on a schedule, re-aggregating summary tables, generating digests or reports. | ||
|
|
||
| For expiring old records, prefer the table-level [`expiration` and `eviction` options](../database/schema.md#table) - record cleanup is built into Harper and does not need a scheduled job. | ||
|
|
||
| ## Configuration | ||
|
|
||
| In your component's `config.yaml`, use the `scheduler` key to declare jobs: | ||
|
|
||
| ```yaml | ||
| scheduler: | ||
| jobs: | ||
| - name: daily-metrics-snapshot | ||
| cron: '0 2 * * *' | ||
| timezone: America/New_York | ||
| handler: ./jobs.js#snapshotMetrics | ||
| - name: sync-exchange-rates | ||
| interval: 15m | ||
| handler: ./jobs.js#syncExchangeRates | ||
| ``` | ||
|
|
||
| Each job entry supports: | ||
|
|
||
| ### `name` | ||
|
|
||
| Type: `string` (required) | ||
|
|
||
| A name for the job, unique within the component. Used in logs and to key the job's run state. | ||
|
|
||
| ### `cron` | ||
|
|
||
| Type: `string` | ||
|
|
||
| A five-field cron expression: minute, hour, day-of-month, month, day-of-week. Supports `*`, lists (`1,15`), ranges (`9-17`), steps (`*/15`, `0-59/5`), month and day names (`JAN`, `MON-FRI`), and the common macros (`@hourly`, `@daily`, `@midnight`, `@weekly`, `@monthly`, `@yearly`). Day-of-week `0` and `7` both mean Sunday. Following the POSIX cron rule, when both day-of-month and day-of-week are restricted, the job runs when either matches. | ||
|
|
||
| There is no seconds field. Expressions that can never fire (such as `0 0 30 2 *` - February 30th) are rejected when the component loads. | ||
|
|
||
| Always quote the expression in YAML: a leading `*` (as in `*/5 * * * *`) is YAML alias syntax and `@daily` starts with a reserved character - unquoted, both fail with confusing YAML parse errors rather than anything about cron. | ||
|
|
||
| Exactly one of `cron` or `interval` is required. | ||
|
|
||
| ### `interval` | ||
|
|
||
| Type: `string` (duration) or `number` (seconds) | ||
|
|
||
| A simple cadence instead of a cron expression: a number of seconds, or a duration string like `90s`, `5m`, `1h`, `1d`. Must be between one second and 365 days. Use this for "every N" maintenance passes that do not align to wall-clock times. | ||
|
|
||
| ### `timezone` | ||
|
|
||
| Type: `string` | ||
|
|
||
| An IANA timezone (for example `America/Chicago`) the cron expression is evaluated in. Defaults to the server's timezone. Only valid with `cron`. | ||
|
|
||
| During daylight-saving transitions: a job scheduled inside the spring-forward gap (a wall-clock time that does not exist that day) runs at the shifted instant rather than being skipped, and a job scheduled inside the fall-back overlap (a wall-clock time that occurs twice) runs once, at the first occurrence. | ||
|
|
||
| ### `handler` | ||
|
|
||
| Type: `string` (required) | ||
|
|
||
| The module and export to invoke, as `<module path>#<named export>`, relative to the component directory - for example `./jobs.js#snapshotMetrics`. Omit the `#...` suffix to use the module's default export. Do not put a space before the `#` (YAML would treat the rest of the line as a comment and silently drop the export name). The module is loaded in your component's environment, so it has access to the same globals (`tables`, `databases`, `server`, and so on) as the rest of your component code. | ||
|
|
||
| A bad handler reference (missing module, missing export, non-function export) fails the component load at deploy time rather than failing silently at the first scheduled fire. | ||
|
|
||
| ## Writing a Handler | ||
|
|
||
| The handler is called with a single context argument and may return a promise: | ||
|
|
||
| ```javascript | ||
| // Nightly snapshot: roll the current state of a table into a dated snapshot record | ||
| export async function snapshotMetrics(context) { | ||
| // context.jobName - the configured job name | ||
| // context.scheduledAt - the occurrence this run is for (Date) | ||
| // context.catchUp - true when this run is making up a missed occurrence | ||
| const day = context.scheduledAt.toISOString().slice(0, 10); | ||
| let total = 0; | ||
| let active = 0; | ||
| for await (const device of tables.Devices.search([])) { | ||
| total++; | ||
| if (device.status === 'active') active++; | ||
| } | ||
| // Keyed by day, so a duplicate delivery of the same occurrence just rewrites | ||
| // the same record - this is what makes the handler idempotent | ||
| await tables.DailyDeviceSnapshot.put({ id: day, total, active }); | ||
| } | ||
|
|
||
| // Scheduled pull from an external API into a Harper table | ||
| export async function syncExchangeRates(context) { | ||
| const response = await fetch('https://api.example.com/rates'); | ||
| if (!response.ok) throw new Error(`rates API responded ${response.status}`); | ||
| const rates = await response.json(); | ||
| await tables.ExchangeRates.put({ id: 'latest', fetchedAt: context.scheduledAt, ...rates }); | ||
| } | ||
| ``` | ||
|
|
||
| **Handlers should be idempotent.** Harper's clustering has no distributed lock, so leadership failover and daylight-saving fall-back can occasionally deliver the same logical occurrence twice. Design handlers so that running twice for one occurrence is harmless. | ||
|
|
||
| Runs of the same job never overlap. Missed time is not queued up, but it is also not always skipped outright: when a run outlasts its cadence, an interval job's next run starts promptly after the previous one finishes (a slow handler can therefore run back-to-back), and a cron job makes up at most its single most recent missed occurrence (delivered with `context.catchUp` set to `true`). | ||
|
|
||
| ## Cluster Behavior | ||
|
|
||
| One node in the cluster - the scheduler leader - runs all scheduled jobs; the others watch. Leader election is automatic and requires no configuration: | ||
|
|
||
| - The leader maintains a heartbeat in the replicated `system` database. If it stops heartbeating (crash, shutdown, partition) for more than five minutes, the next node in line promotes itself; with default timings, failover completes within about six and a half minutes. | ||
| - A restarting node defers to an actively heartbeating leader, so leadership is stable across deploys and restarts. | ||
| - On a single-node instance, that node simply runs the jobs. | ||
|
|
||
| ### Missed Occurrences | ||
|
|
||
| If the most recent occurrence of a cron job was missed - the leader was down, a failover was in progress, or daylight-saving time skipped the slot - the scheduler fires one catch-up run for it (with `context.catchUp` set to `true`). Only the single most recent missed occurrence is made up, not every occurrence missed during an outage. Interval jobs resume their cadence from their last recorded run. | ||
|
|
||
| A newly deployed job waits for its first scheduled time; it does not fire immediately on deploy. | ||
|
|
||
| ### Run State | ||
|
|
||
| Each job's last run time, status, duration, and last error (if any) are recorded in the `hdb_scheduler_state` system table, alongside the leader lease. This state replicates across the cluster so a newly promoted leader knows what has already run. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit, non-blocking:
tables.Devices.search([])is valid (Table.tsaccepts a bare array asconditionsdirectly), butresource-api.mdonly documents thesearch({})object form for an unconditioned scan. Consider usingsearch({})here to match the idiom taught elsewhere, so a reader who cross-references resource-api.md finds the exact form used in this example.