You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
cron: the heartbeat renewal is awaited inside the select that polls the run, so the run pauses for it #375
A cron worker drives a running occurrence in a select! over the work, the fence deadline and the heartbeat ticker. The ticker arm awaits renew() — two SQLite UPDATEs — inside the arm body (cron/worker.rs:775-783), so while a renewal is in flight the work future is not polled: the run pauses for the renewal's duration, every heartbeat_interval_secs.
Normally sub-millisecond to a few ms. Under state-database write contention (a busy node, a tmpfs-backed SQLite shared with the scheduler's inserts), it is up to busy_timeout or the fence deadline — a stall inside a turn of a latency-sensitive loop, charged to whatever the run was awaiting (a model_infer deadline counts wall time from the call).
Proposal
Renew concurrently with the work: spawn the renewal (or keep it as a separate pinned future in the select!) so the work keeps being polled, and only the renewal's outcome — lost or fenced — interrupts it.
Fixed in 61fa233 on branch fix/issues-369-377 (not yet pushed; it lands after PR #368). The lease renewal is now its own branch of the heartbeat select!, polled beside the work with at most one in flight, so a slow renewal no longer pauses the run. A test runs 20 one-second steps against 10-second renewals and needs exactly 20 seconds.
Activity
codetiger commentedon Oct 1, 2026
Fixed in 61fa233 on branch
fix/issues-369-377(not yet pushed; it lands after PR #368). The lease renewal is now its own branch of the heartbeatselect!, polled beside the work with at most one in flight, so a slow renewal no longer pauses the run. A test runs 20 one-second steps against 10-second renewals and needs exactly 20 seconds.fix: the five Orion-side issues from the turn-loop profiling (#369, #370