Skip to content

cron: the heartbeat renewal is awaited inside the select that polls the run, so the run pauses for it #375

Closed

Description

@codetiger

Summary

A cron worker drives a running occurrence in a select! over the work, the fence deadline and the heartbeat ticker. The ticker arm awaits renew() — two SQLite UPDATEs — inside the arm body (cron/worker.rs:775-783), so while a renewal is in flight the work future is not polled: the run pauses for the renewal's duration, every heartbeat_interval_secs.

Normally sub-millisecond to a few ms. Under state-database write contention (a busy node, a tmpfs-backed SQLite shared with the scheduler's inserts), it is up to busy_timeout or the fence deadline — a stall inside a turn of a latency-sensitive loop, charged to whatever the run was awaiting (a model_infer deadline counts wall time from the call).

Proposal

Renew concurrently with the work: spawn the renewal (or keep it as a separate pinned future in the select!) so the work keeps being polled, and only the renewal's outcome — lost or fenced — interrupts it.

Activity

  1. codetiger commented on Oct 1, 2026

    @codetiger
    MemberAuthor

    Fixed in 61fa233 on branch fix/issues-369-377 (not yet pushed; it lands after PR #368). The lease renewal is now its own branch of the heartbeat select!, polled beside the work with at most one in flight, so a slow renewal no longer pauses the run. A test runs 20 one-second steps against 10-second renewals and needs exactly 20 seconds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions