Skip to content

RFC: making sccache-dist work for OpenEmbedded/Yocto host-mode distributed builds #2775

Description

@jetm

I've been getting sccache-dist working for OpenEmbedded/Yocto host-mode
distributed builds, and the work has grown well past a single reviewable change.
I originally opened it as #2750, which I've now closed and am splitting into
small, focused PRs landed in waves. This issue is the discussion anchor for that
series, especially the scheduler work, before those PRs land.

What the series covers

Bug fixes (independent, landing first):

  • A server-side panic when a finished compile reports neither exit code nor
    signal.
  • Local fallback when a distributed compile fails or drops an output, instead of
    failing the whole build.
  • gcc/rust distributed-compile correctness: force-local for feature probes and
    PCH generation; ship the rust sysroot std, target spec, and rlibs; toolchain
    packaging fixes for relocated and split-sysroot cross toolchains.

Scheduler and server reliability under high concurrency (a base-first stack):

  • Prompt removal of stopped servers; topology-aware, deterministic routing;
    idempotent job-state updates and lost/stale-job reaping; liveness-lease
    reaping; restart-safe job accounting.
  • Per-toolchain preparation locks so concurrent compiles stop convoying, and
    HTTP connection reuse.

Two compatibility surfaces I want to flag up front

  1. The liveness-lease reaper adds an active_jobs field to the heartbeat
    request, which changes the bincode wire format: scheduler and build servers
    must run matching binaries. Old-server to new-scheduler fails loudly;
    new-server to old-scheduler silently drops the field. I'm happy to gate this
    behind a protocol version if you'd prefer.
  2. Connection reuse adds two DistConfig keys (with serde(default), so old
    configs keep parsing) and a small arc-swap dependency.

Questions for maintainers

  • The scheduler routing work introduces some new policy surface (topology-aware
    placement, a dynamically derived colocated de-weight, advertised core counts).
    It is opt-in via config with defaults that preserve current behavior. Is that
    direction welcome, or would you rather shape it before the PRs land?
  • There is a set of load-diagnosis instrumentation I used to find these issues.
    I can submit it, fold the useful parts into the PRs that needed them, or drop
    it. Preference?

I'll link the individual PRs here as they open. Happy to adjust the split,
ordering, or scope to match how you'd like to review this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions