You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
cron: a saturated forbid+slots channel queues a pending row per tick that only drains when the node goes idle #377
cron_cleanup deletes completed, failed, skipped_misfire and skipped_singleton occurrences past [trace_queue] retention_hours, but — deliberately — never a pending one, since "a pending occurrence older than the retention window is a backlog, not history" (storage/repositories/cron.rs:1235-1237). And a due tick always inserts a row, whether or not a slot or a worker is free (cron/scheduler.rs:204-226).
On a node whose forbid + slots channel is saturated, only the spare cron workers claim rows — one claim per poll_interval_ms each — while the schedule keeps inserting. With workers = slots + 1 and every slot busy, a schedule faster than the one spare worker drains (e.g. every second, plus other channels' ticks) grows the pending backlog for as long as the node is busy — and since a pending row is (rightly) never cleaned up, the backlog only drains once the node goes idle. On a small state database (a runner's SQLite on a 64 MiB tmpfs) that is an outage in waiting; today it simply forbids schedules faster than ~5 s on such a node.
Proposals (any one helps)
A due tick for a forbid channel whose slots are all held could be recorded skipped_singleton at insert time (or coalesced into one row), rather than queued as pending.
Settle a pending row that a worker claims after it has aged past misfire_grace_secs as skipped_misfire (misfire is judged only at insert today, channel/cron.rs:777-836), so a backlog of stale ticks collapses into terminal rows the cleanup can remove.
A first-class "run again now" for a cron channel from inside its own workflow (today: http_call to POST /admin/channels/{id}/trigger on the node's own admin API, which also answers 409 when a scheduled row owns the same second, and writes an audit row per call), so a busy loop can chain runs without a fast schedule at all.
Fixed in 61fa233 on branch fix/issues-369-377 (not yet pushed; it lands after PR #368) (first proposal). The reconciler checks the key's live slot holds when a tick comes due and writes a forbid occurrence skipped_singleton straight away if every slot is held, so nothing is queued pending that could only be skipped. Rows written already settled now carry completed_at.
Activity
codetiger commentedon Oct 1, 2026
Fixed in 61fa233 on branch
fix/issues-369-377(not yet pushed; it lands after PR #368) (first proposal). The reconciler checks the key's live slot holds when a tick comes due and writes aforbidoccurrenceskipped_singletonstraight away if every slot is held, so nothing is queuedpendingthat could only be skipped. Rows written already settled now carrycompleted_at.fix: the five Orion-side issues from the turn-loop profiling (#369, #370
chore(release): 1.12.0