github-app-plan.md

husk as a GitHub App — implementation plan

Turn huskd from a single-repo, PAT-authenticated runner autoscaler into a GitHub App installable on arbitrary orgs/repos, gated by a huskd-side allowlist, and do it in the same pass as the async / centralized-poller refactor so the new code is written once in its final form.

Settled decisions

• Restriction lives in huskd, not GitHub. GitHub can't restrict a public App to a set of orgs, so the App is installable by any account and huskd decides who it serves. huskd holds the App private key and runs the VMs, so an install it doesn't recognize simply gets no runners.
• Each pool names the one target it serves (revised 2026-07-20; originally a two-list [access] allowlist fanned out over all pools):
  • target = { org = "acts-project", group = "husk" } → org-level runners.
  • target = { repo = "paulgessinger/husk-test" } → repo-level runners for exactly that owner/repo, nothing else that owner owns.
  • Defense in depth: the install's granted-repo set and huskd's config must both agree before a repo is served.
  • group is nested in the target table because runner groups are an org-only concept — the schema makes "group on a repo target" unrepresentable.
• Hybrid scope. Org-level is the scalable default (one poll + one warm pool per org, existing busy + min_ready math, scales to any repo count). Repo-level (today's code path) is the fallback for personal-account projects. Personal accounts have no org-level runners, so this is the only way to ever support paulgessinger/* projects.
• Delivery = JIT ephemeral runners (unchanged from today), via App installation tokens instead of a PAT. The generate-jitconfig / list / delete / reap endpoints are identical; only the bearer token and the /orgs|/repos path prefix change.
• Poll, don't webhook (for now). huskd runs in a restricted network with no inbound reachability; the whole design is poll-by-design. Discovery polls GET /app/installations; demand polls the runner list per target. Webhooks are a deferred, non-blocking accelerant (Phase 4).
• Async-first sequencing. The async / centralized-poller refactor lands before the App migration so the token provider, discovery, and scope branching are born async and written exactly once (see rationale below).

The conceptual shift

• Reconcile unit: pool → (target, pool), where a target is tagged org:<login> or repo:<owner/name>. A pool is a runner type (labels + backend); a target is a place to put runners.
• Demand-signal seam: reconcile consumes desired(target, pool) from an in-memory registry; it does not call GitHub inline. One centralized poller is the only producer today; a webhook handler becomes a second producer later.
• Runner-group gotcha: runner_group_id is not portable across orgs. Config moves to a runner-group name, resolved to an ID per target (GET /orgs/{org}/actions/runner-groups), falling back to Default (1). Repo / personal path ignores groups.
• Name isolation (unique vm_prefix / labels, already required because runner APIs are repo-wide) is sufficient as-is now that a pool maps to exactly one target: load_configs enforces unique pool names and prefixes, and no target-folded renaming is needed. Ownership is additionally enforced at the backend listing (husk-pool metadata) — see targets-and-capacity.md.

Why async before the App (churn rationale)

Migrating to the App on the current threaded / requests model first would mean writing the token provider and discovery synchronously and then porting them to async — double work on precisely the newest code. Async-first means they're born async. The only re-editing is the GitHub client (keying → async → auth/paths), and those three touches are orthogonal, not rewrites.

Concern	Written / rewritten	Times touched
Target + demand seam	Phase 0	1 (kept forever)
Async port of client + reconcile	Phase 1	1 (on trivial domain)
Token provider, discovery, scope branching	Phases 2–3	1 each, born async
Reconcile loop	untouched after Phase 1	seam absorbs App changes

Target config schema (end state)

Clean cutover — no back-compat, no dual-form parsing. Rewrite config.example.toml and drop repo / pat / pat_path / pat_env.

[github]                                    # App identity replaces repo + pat
app_id = 123456
private_key_path = "/etc/husk/husk-app.pem" # or HUSK_GITHUB__PRIVATE_KEY

[[pool]]                                     # each pool names the ONE target it serves
name   = "gpu"
target = { org = "acts-project", group = "husk" }   # org-level runners
# ...backend + runner config as today...

[[pool]]
name   = "test"
target = { repo = "paulgessinger/husk-test" }       # that repo only

Superseded (2026-07-20): the design above originally had an [access] allowlist fanned out over every pool. It was replaced by explicit per-pool binding, because warm capacity cannot be shared across targets — a JIT runner belongs to exactly one org/repo — so fan-out silently multiplied min_ready and over-subscribed scarce hardware, while org scope already covers the common "many repos" case with one target. See targets-and-capacity.md.

Phases

Each phase is independently shippable and testable; you can stop after any phase and have a working system.

Phase 0 — Shared abstractions, zero behavior change

(still PAT, still threads, still requests, one repo)

• Add a Target type (org:<login> / repo:<owner/name>); re-key reconcile pool → (target, pool) with a single static target derived from today's repo (cardinality 1).
• Add the demand-signal seam: reconcile reads desired(target, pool) from an in-memory registry instead of calling GitHub inline. The same inline poll fills the registry behind the interface.
• Ships: identical behavior, verifiable against current husk.
• Churn: none — every abstraction here survives to the end. May be folded into Phase 1; kept separate because doing the re-keying in the familiar sync model de-risks it.

Phase 1 — Async + centralized poller (target-architecture refactor) ✅ SHIPPED

(still PAT, still one target)

• Port the GitHub client requests → httpx, sync → async; replace the daemon-thread deadline hack with asyncio.wait_for. Done — httpx per-op timeout + wait_for wall-clock backstop. requests dropped as a direct dep.
• One centralized async poller task fills the SnapshotRegistry; reconcile becomes async tasks per (target, pool) reading the registry. Single async Quart process. Done — husk/poller.py; MultiPoolController.run(stop) spawns one asyncio task per pool; Quart + poller + reconcile share one event loop.
• Backends stay synchronous by design: Controller.tick() pushes every backend call through asyncio.to_thread, so we keep openstacksdk/libvirt-python rather than hand-rolling REST. That wrapping is load-bearing — an unwrapped blocking call stalls the whole loop. (openstacksdk is sync even though Nova is HTTP; native-async Nova would mean reimplementing keystone auth + microversions, so it's deferred as a possible later optimization.)
• File state was already gone; nothing to drop.

Runner-snapshot freshness policy (new, deliberate — carries into Phases 2–4): a failed poll keeps the last good snapshot so a GitHub blip can't stall reconciliation, while the controller refuses a snapshot older than RUNNER_SNAPSHOT_MAX_AGE_S (180s) and fail-safes the tick. That age check is what preserves the old "GitHub is down ⇒ take no action" guarantee now that the inline list_runners() raise is gone.

• Ships: behavior identical, now async — 298 unit tests plus an end-to-end smoke of the real _serve composition (poller polling, both pools reconciling independently, all endpoints serving, loop responsive, clean SIGTERM).
• Churn: the one big mechanical rewrite, done while the domain is one PAT target so correctness is easy to check.

Phase 2 — GitHub App auth (swap PAT → App) ✅ SHIPPED

(targets still static/explicit — no discovery yet)

Shipped as described below, plus:

• husk/ghhttp.py holds the shared API base / two-layer timeout / error type, so appauth and github share plumbing without an import cycle.
• Reconcile is now genuinely (target, pool): one Controller per pair. With

  1 target the pool name and vm_prefix fold in the target (gpu@acts-project, husk-gpu-acts-project) so names can't collide; with a single target both are left untouched, because changing vm_prefix would orphan every running VM.

• Runner-group resolution degrades rather than fails: an unknown name (or a failed listing) falls back to Default/1 — huskd serves orgs it does not administer, so a group named in its own config may simply not exist there.
• One InstallationTokenProvider process-wide, serialized on a lock so N pools sharing a target don't stampede the mint endpoint.
• huskctl reap/recycle are now target-scoped: reap iterates every target; recycle unions runner listings across targets for busy detection.
• Verified by 321 unit tests plus an end-to-end smoke of the real _serve composition on App auth (org + repo targets: JWT→installations→per-installation tokens, /orgs vs /repos branching, group husk→7 on org and absent on repo, per-target name isolation, clean shutdown).
• InstallationTokenProvider: sign RS256 JWT (10-min exp, in-memory) from app_id + PEM; exchange for per-installation tokens (POST /app/installations/{id}/access_tokens), cache per installation_id, refresh at ~55 min or on 401. Written once, async.
• Config cutover: App identity replaces repo + pat. A temporary explicit target key (e.g. [access] targets = ["org:acts-project"]) stands in for discovery.
• Scope branching in the client paths (/orgs/… vs /repos/…); runner-group name → ID resolution per target.
• Ships: App-authenticated runners against a known target.
• Churn: ~10 lines of throwaway (the temp targets key), a deliberate cost to isolate auth bugs from discovery bugs. Optional to merge with Phase 3.

Phase 3 — Dynamic discovery + allowlist ✅ SHIPPED

Shipped as described below, plus:

• husk/discovery.py holds Allowlist (validating + case-insensitive matching that preserves the operator's spelling, since vm_prefix derives from it) and TargetDiscovery. The repo listing is skipped entirely for an org-only allowlist — one fewer API call per installation per sweep.

• Failure policy, mirroring the runner poller's. A sweep reports whether it was complete; a failed sweep changes nothing at all, and a partial one (some installation's repo listing failed) may only add targets. Absence from an incomplete result is never evidence of removal — otherwise a GitHub 500 would tear down live runners.

• Removal drains, it doesn't destroy. A de-allowlisted/uninstalled target stops reconciling immediately, then each sweep deregisters and destroys its idle slots; busy slots are left running and retried, so an in-flight job is never killed. A target that reappears mid-drain is revived (same Controller, slots intact) rather than rebuilt. A backend that can't list holds the drain open rather than being read as "nothing to clean up".

• Per-target naming keys off the allowlist size, not the discovered set: the discovered set moves as people install/uninstall, and a vm_prefix that changed under a running slot would orphan it.

• Pagination added to /app/installations and /installation/repositories — huskd is installable by any account, so page-1 truncation would silently not serve the 31st install.

• Fixed a Phase 2 regression: config reload matched pools by backend.name, which the gpu@target fold broke. Reload now maps a live unit back to its [[pool]] on the base name, and @ is reserved in configured pool names.

• huskctl reap/recycle discover their targets too (no config target list to read any more).

• Verified by 349 unit tests plus an end-to-end smoke of the real _serve composition driving an actual lifecycle mid-flight: org install → unit appears; repo install added → second unit spawns with no restart; a non-allowlisted org install → ignored; a granted-but-not-allowlisted repo → never minted against; repo install removed → unit stops, runner deregistered, slot destroyed, surviving target untouched.

• Discovery poller: GET /app/installations → for each install read account.login; if in allowed_orgs emit an org target; regardless, GET /installation/repositories ∩ allowed_repos → emit repo targets.

• Drive reconcile-task lifecycle: spawn on new target, reap (deregister runners + stop tasks) on removed target.

• Two-list allowlist config replaces the temp targets key.

• Ships: full "install on arbitrary org/repo, gated by huskd."

• Churn: none — feeds machinery already built; only makes the target set dynamic.

Phase 4 — Webhooks (deferred, unblocked)

• POST /webhook + X-Hub-Signature-256 verification as a second producer nudging the registry. Poll stays as the backstop even after this lands. Drops in because the seam (Phase 0) and async loop (Phase 1) already exist.

GitHub App setup (one-time, manual, outside code)

• Create the App owned by whichever org, installable on "Any account."
• Permissions: Organization self-hosted runners: write (org path); Administration: write on repos (repo / personal fallback); Metadata: read; Actions: read.
• Subscribe to no events yet (webhooks deferred). Download the private-key PEM.

Validation

• Unit: JWT signing, token cache expiry/refresh, allowlist filtering (org + repo lists), org-vs-repo path builder, runner-group name → ID resolution.
• Live (mirrors the POC discipline): install the App on acts-project, confirm discovery → org-level JIT mint → runner appears → job runs → reaping; then a personal-account install of paulgessinger/husk-test to exercise the repo-level fallback.

Open items

• Whether to auto-decline (DELETE /app/installations/{id}) non-allowlisted installs or just ignore them. Settled: ignore. Declining is destructive and irreversible from huskd's side; someone experimenting with a public App should not have their install silently deleted.
• Merge Phase 2 + Phase 3 to avoid the temp targets scaffold. Kept separate; the scaffold cost ~10 lines and isolated auth bugs from discovery bugs.
• serve_targets per-pool mapping — still open, still defaulted: every pool serves every discovered target. Worth revisiting once there's a pool that shouldn't be offered to every org (e.g. an expensive GPU pool).
• Repo scope was reconsidered and kept (2026-07-20): dropping it would remove ~200 mostly-mechanical lines and the partial-sweep rule, but it is already written and validated, and it is the only way to serve a personal account — personal accounts have no org-level runners.