Skip to content

feat(container-runner): setup instance memory and cpu usage logger - #5549

Merged
abcxff merged 9 commits into
mainfrom
stack/feat-container-runner-setup-instance-memory-and-cpu-usage-logger-xumpnzoy
Aug 11, 2026
Merged

feat(container-runner): setup instance memory and cpu usage logger#5549
abcxff merged 9 commits into
mainfrom
stack/feat-container-runner-setup-instance-memory-and-cpu-usage-logger-xumpnzoy

Conversation

@abcxff

@abcxff abcxff commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@claude

claude Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Review

Nice addition — the cgroup v2 / /proc fallback split and the opt-in gating are both sound, and the log fields follow the structured-logging convention. A few things worth tightening:

1. detect_source() re-probes the filesystem on every actor start, uncached (moderate)
container-runner/src/actor.rs calls crate::monitor::sampling_source() unconditionally in on_start (an async fn), which calls detect_source() and does up to 2 synchronous std::fs::read_to_string calls on the tokio executor thread — every time any actor starts on the instance — regardless of whether RIVET_LOG_RESOURCE_USAGE is even set. Since cgroup v2 vs /proc availability is fixed for the lifetime of the container, this should be detected once and cached (e.g. std::sync::OnceLock<Option<Source>>) rather than re-probed per actor start. As written it adds unconditional sync I/O to the actor-startup hot path for a feature that is off by default.

2. active_actor_ids() uses retain_async for a read-only scan (minor)
container-runner/src/main.rs:

CHILDREN.retain_async(|actor_id, _| {
    ids.push(actor_id.clone());
    true
}).await;

This codebase already has an idiomatic scc read-only-scan pattern, iter_async, used for the exact same "collect keys" shape elsewhere (e.g. rivetkit-rust/packages/client/src/connection.rs:287, engine/packages/universalpubsub/src/driver/memory/mod.rs:181). retain_async signals mutation intent to a reader and is heavier than a pure read scan per sccs contract. Prefer iter_async here to match convention and avoid the implied write-lock semantics, especially since this now runs every 500ms while any actor is up and contends with children().insert_async / remove_async during actor spawn and stop.

3. No test coverage for the new parsing/calculation logic (minor)
monitor.rs adds several pure functions (read_meminfo_kb, read_proc_cpu_busy_usec, read_proc_cpu_count, read_cgroup_cpu_limit_cores, the cpu.max / cpu.stat parsers) that are easy to unit test by feeding static file contents, but none were added. Given how fiddly /proc/stat field parsing is, these are prime candidates for direct unit tests, possibly with a small refactor to take a &str instead of a hardcoded path so tests can inject fixture content.

4. Possible double-count in read_proc_cpu_busy_usec (low severity, accuracy)

let total: u64 = values.iter().sum();
let idle = values[3] + values.get(4).copied().unwrap_or(0);

total sums all fields including guest / guest_nice, which on Linux are already included in user / nice. Under guest-heavy workloads this double-counts and can push cpu_pct above the true value. Probably fine for the gVisor fallback case this is really targeting, but worth a comment noting the known imprecision, or excluding guest / guest_nice from total.

5. New env var bypasses the existing self-documentation mechanism (nit)
Other instance-tunable knobs in main.rs (RIVET_STOP_GRACE_SECS, RIVET_READINESS_TIMEOUT_SECS, etc.) are wired through claps #[arg(env = "...")], so they show up in --help. RIVET_LOG_RESOURCE_USAGE is read directly via std::env::var in monitor.rs and is not discoverable that way or mentioned in the README. Not required, but worth considering for consistency.

Other notes

  • Log volume: at 500ms and one line per running actor, this can get noisy on multi-actor instances over a long session. Seems intentional for the "see the OOM coming" use case, just flagging in case the interval should be configurable too.
  • Security / trust boundaries: no concerns. This only reads local pseudo-files (/sys/fs/cgroup, /proc) and is opt-in via env var, no externally-controlled input involved.
  • MissedTickBehavior::Skip plus the guarded elapsed.is_zero() / saturating_sub usage in the CPU delta calc look correct, no underflow or panic risk there.

@abcxff
abcxff force-pushed the stack/feat-container-runner-setup-instance-memory-and-cpu-usage-logger-xumpnzoy branch from 886a208 to 33d70a3 Compare August 10, 2026 17:16
@abcxff
abcxff force-pushed the stack/fix-rivetkit-keep-actor-event-loop-alive-through-shutdown-state-serialization-losmyvnt branch from 40d03bf to 13cd549 Compare August 10, 2026 17:16
@abcxff
abcxff force-pushed the stack/fix-rivetkit-keep-actor-event-loop-alive-through-shutdown-state-serialization-losmyvnt branch from 13cd549 to fbd0e22 Compare August 10, 2026 19:03
@abcxff
abcxff force-pushed the stack/feat-container-runner-setup-instance-memory-and-cpu-usage-logger-xumpnzoy branch 2 times, most recently from 45f065f to c0e196d Compare August 10, 2026 20:35
@abcxff
abcxff force-pushed the stack/fix-rivetkit-keep-actor-event-loop-alive-through-shutdown-state-serialization-losmyvnt branch from fbd0e22 to 7df3e38 Compare August 10, 2026 20:35
@abcxff
abcxff force-pushed the stack/feat-container-runner-setup-instance-memory-and-cpu-usage-logger-xumpnzoy branch from c0e196d to 8d63009 Compare August 10, 2026 22:09
@abcxff
abcxff force-pushed the stack/fix-rivetkit-keep-actor-event-loop-alive-through-shutdown-state-serialization-losmyvnt branch from 7df3e38 to 5c46256 Compare August 10, 2026 22:09
@abcxff
abcxff force-pushed the stack/feat-container-runner-setup-instance-memory-and-cpu-usage-logger-xumpnzoy branch from 8d63009 to cabfb3c Compare August 11, 2026 13:39
@abcxff
abcxff force-pushed the stack/fix-rivetkit-keep-actor-event-loop-alive-through-shutdown-state-serialization-losmyvnt branch from 5c46256 to b8b0fd3 Compare August 11, 2026 13:39
@abcxff
abcxff force-pushed the stack/feat-container-runner-setup-instance-memory-and-cpu-usage-logger-xumpnzoy branch from cabfb3c to 98b0536 Compare August 11, 2026 14:24
@abcxff
abcxff force-pushed the stack/fix-rivetkit-keep-actor-event-loop-alive-through-shutdown-state-serialization-losmyvnt branch from b8b0fd3 to f8766f0 Compare August 11, 2026 14:24
@abcxff
abcxff force-pushed the stack/feat-container-runner-setup-instance-memory-and-cpu-usage-logger-xumpnzoy branch from 98b0536 to d0608e2 Compare August 11, 2026 17:22
@abcxff
abcxff force-pushed the stack/fix-rivetkit-keep-actor-event-loop-alive-through-shutdown-state-serialization-losmyvnt branch from f8766f0 to 4451538 Compare August 11, 2026 17:22
@abcxff
abcxff changed the base branch from stack/fix-rivetkit-keep-actor-event-loop-alive-through-shutdown-state-serialization-losmyvnt to main August 11, 2026 17:24
@abcxff
abcxff merged commit d0608e2 into main Aug 11, 2026
3 of 7 checks passed
@abcxff
abcxff deleted the stack/feat-container-runner-setup-instance-memory-and-cpu-usage-logger-xumpnzoy branch August 11, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant