I've been getting sccache-dist working for OpenEmbedded/Yocto host-mode
distributed builds, and the work has grown well past a single reviewable change.
I originally opened it as #2750, which I've now closed and am splitting into
small, focused PRs landed in waves. This issue is the discussion anchor for that
series, especially the scheduler work, before those PRs land.
What the series covers
Bug fixes (independent, landing first):
- A server-side panic when a finished compile reports neither exit code nor
signal.
- Local fallback when a distributed compile fails or drops an output, instead of
failing the whole build.
- gcc/rust distributed-compile correctness: force-local for feature probes and
PCH generation; ship the rust sysroot std, target spec, and rlibs; toolchain
packaging fixes for relocated and split-sysroot cross toolchains.
Scheduler and server reliability under high concurrency (a base-first stack):
- Prompt removal of stopped servers; topology-aware, deterministic routing;
idempotent job-state updates and lost/stale-job reaping; liveness-lease
reaping; restart-safe job accounting.
- Per-toolchain preparation locks so concurrent compiles stop convoying, and
HTTP connection reuse.
Two compatibility surfaces I want to flag up front
- The liveness-lease reaper adds an
active_jobs field to the heartbeat
request, which changes the bincode wire format: scheduler and build servers
must run matching binaries. Old-server to new-scheduler fails loudly;
new-server to old-scheduler silently drops the field. I'm happy to gate this
behind a protocol version if you'd prefer.
- Connection reuse adds two
DistConfig keys (with serde(default), so old
configs keep parsing) and a small arc-swap dependency.
Questions for maintainers
- The scheduler routing work introduces some new policy surface (topology-aware
placement, a dynamically derived colocated de-weight, advertised core counts).
It is opt-in via config with defaults that preserve current behavior. Is that
direction welcome, or would you rather shape it before the PRs land?
- There is a set of load-diagnosis instrumentation I used to find these issues.
I can submit it, fold the useful parts into the PRs that needed them, or drop
it. Preference?
I'll link the individual PRs here as they open. Happy to adjust the split,
ordering, or scope to match how you'd like to review this.
I've been getting sccache-dist working for OpenEmbedded/Yocto host-mode
distributed builds, and the work has grown well past a single reviewable change.
I originally opened it as #2750, which I've now closed and am splitting into
small, focused PRs landed in waves. This issue is the discussion anchor for that
series, especially the scheduler work, before those PRs land.
What the series covers
Bug fixes (independent, landing first):
signal.
failing the whole build.
PCH generation; ship the rust sysroot std, target spec, and rlibs; toolchain
packaging fixes for relocated and split-sysroot cross toolchains.
Scheduler and server reliability under high concurrency (a base-first stack):
idempotent job-state updates and lost/stale-job reaping; liveness-lease
reaping; restart-safe job accounting.
HTTP connection reuse.
Two compatibility surfaces I want to flag up front
active_jobsfield to the heartbeatrequest, which changes the bincode wire format: scheduler and build servers
must run matching binaries. Old-server to new-scheduler fails loudly;
new-server to old-scheduler silently drops the field. I'm happy to gate this
behind a protocol version if you'd prefer.
DistConfigkeys (withserde(default), so oldconfigs keep parsing) and a small
arc-swapdependency.Questions for maintainers
placement, a dynamically derived colocated de-weight, advertised core counts).
It is opt-in via config with defaults that preserve current behavior. Is that
direction welcome, or would you rather shape it before the PRs land?
I can submit it, fold the useful parts into the PRs that needed them, or drop
it. Preference?
I'll link the individual PRs here as they open. Happy to adjust the split,
ordering, or scope to match how you'd like to review this.