fix(metrics): prevent GitHub API calls from hanging forever

[whywaita]

Summary

Fixes GET /metrics hanging and never returning a response. On a production pod, /metrics returned nothing after 60s (curl localhost:80/metrics → http=000 time_total=60s size=0).

Root cause

The /metrics Prometheus collector performs live GitHub API calls (Apps.ListInstallations, Actions.ListRepositoryWorkflowRuns, installation-token refresh) synchronously and waits on them with wg.Wait(). The GitHub *http.Client has no Timeout (and http.DefaultTransport has no ResponseHeaderTimeout), so when a connection to the GitHub API stalls, the request blocks indefinitely and GET /metrics hangs.

Worse, responseCache is only populated on a successful call, so while stalled the two Prometheus scrapers and the starter loop keep re-issuing the same live API calls, which sustains the stall (a self-reinforcing loop).

Evidence from the running pod (last 15 min)

Metric	Count
GET /metrics	273
scrape failures name: github	283
scrape failures datastore / memory	0 / 0
context canceled	566

get installations from GitHub, page: 0, now all installations: 0   # always stuck at page 0 / 0 results, never completes
get workflow runs from GitHub, page: 0, now all runners: 0         # same

The datastore / memory scrapers had 0 failures, api.github.com/zen from the pod responded in ~190ms, and the pod had only 15 TCP sockets / 12 threads (no exhaustion). The only thing stuck was the github scraper waiting on the GitHub App API.

Changes

• pkg/gh/github.go: give every GitHub *http.Client a Timeout (10s) via a new newGitHubHTTPClient helper, so no single request can hang forever. This also protects the starter loop and runner deletion (which were hanging the same way, e.g. unexpected EOF).
• pkg/gh/installation.go: fix listInstallations / listAppsInstalledRepo calling the underlying _list* function twice on a cache miss (the first result was cached and then discarded), which doubled GitHub API load.

Note: the /metrics handler intentionally keeps r.Context() and does not add its own deadline — the scrape timeout is the scraper's (Prometheus) responsibility.

• cmd/server/cmd.go: import net/http/pprof. The server already listens on localhost:6060 and sets SetBlockProfileRate / SetMutexProfileFraction, but the pprof handlers were never registered, so every /debug/pprof/* endpoint returned 404. This makes pprof usable for future investigations (it was unavailable during this one).

Follow-up (separate PR)

This PR stops the hang via timeouts, which is a mitigation. The longer-term fix is to stop doing synchronous live API calls inside the collector and instead refresh in the background and serve cached values (a Prometheus collector should not block on external I/O). That is a larger behavioral change, so it is kept separate.

Testing

• go build ./... and go vet ./pkg/gh/... ./pkg/web/... ./cmd/server/... pass.
• go test ./pkg/gh/... passes (pkg/web requires Docker via dockertest and was not run in this environment).

🤖 Generated with Claude Code