Skip to content

cron: a saturated forbid+slots channel queues a pending row per tick that only drains when the node goes idle #377

Closed

Description

@codetiger

Summary

cron_cleanup deletes completed, failed, skipped_misfire and skipped_singleton occurrences past [trace_queue] retention_hours, but — deliberately — never a pending one, since "a pending occurrence older than the retention window is a backlog, not history" (storage/repositories/cron.rs:1235-1237). And a due tick always inserts a row, whether or not a slot or a worker is free (cron/scheduler.rs:204-226).

On a node whose forbid + slots channel is saturated, only the spare cron workers claim rows — one claim per poll_interval_ms each — while the schedule keeps inserting. With workers = slots + 1 and every slot busy, a schedule faster than the one spare worker drains (e.g. every second, plus other channels' ticks) grows the pending backlog for as long as the node is busy — and since a pending row is (rightly) never cleaned up, the backlog only drains once the node goes idle. On a small state database (a runner's SQLite on a 64 MiB tmpfs) that is an outage in waiting; today it simply forbids schedules faster than ~5 s on such a node.

Proposals (any one helps)

  • A due tick for a forbid channel whose slots are all held could be recorded skipped_singleton at insert time (or coalesced into one row), rather than queued as pending.
  • Settle a pending row that a worker claims after it has aged past misfire_grace_secs as skipped_misfire (misfire is judged only at insert today, channel/cron.rs:777-836), so a backlog of stale ticks collapses into terminal rows the cleanup can remove.
  • A first-class "run again now" for a cron channel from inside its own workflow (today: http_call to POST /admin/channels/{id}/trigger on the node's own admin API, which also answers 409 when a scheduled row owns the same second, and writes an audit row per call), so a busy loop can chain runs without a fast schedule at all.

Activity

  1. codetiger commented on Oct 1, 2026

    @codetiger
    MemberAuthor

    Fixed in 61fa233 on branch fix/issues-369-377 (not yet pushed; it lands after PR #368) (first proposal). The reconciler checks the key's live slot holds when a tick comes due and writes a forbid occurrence skipped_singleton straight away if every slot is held, so nothing is queued pending that could only be skipped. Rows written already settled now carry completed_at.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions