Repository navigation
fix(installer): find the latest version without the API rate limit - #288
Conversation
Unpinned installs failed with "Could not determine CLI version" in Upsun builds, very likely because unauthenticated GitHub API requests are rate limited per IP address, and build hosts share their IPs. The error was hidden, so the log did not show the cause. - Read a GitHub token from COMPOSER_AUTH and ~/.global/auth.json (the token Upsun provides in project containers), as the legacy PHP installer did, as well as from GITHUB_TOKEN and the gh CLI. - Check the token against /rate_limit, which does not count against the limit, and say when a token is ignored. - If the API request fails, log the error and fall back to the redirect of https://github.com/upsun/cli/releases/latest, which is not subject to the API rate limit. Refs #73 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnUBGurqFyfoRBS7aXYqDR
Make the GitHub URLs overridable (RELEASES_URL, RELEASES_API_URL), and add integration tests that run installer.sh against an httptest server: the token sources and their precedence, invalid tokens, the rate-limit fallback to the latest release redirect, and failures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnUBGurqFyfoRBS7aXYqDR
There was a problem hiding this comment.
Warning
Changes suggested — 🟡 2 warnings · 🔵 1 minor point
🔍 Full review · 3 files reviewed
Verification
${RELEASES_API_URL%%/repos/*}/rate_limitresolves tohttps://api.github.com/rate_limitfor the default URL, and to/api/rate_limiton the fake server.GITHUB_TOKENdefaults to empty at line 35, so the new[ -n "${GITHUB_TOKEN}" ]checks are safe underset -eu;COMPOSER_AUTHandHOMEare read with:-defaults.latest_tag_from_redirectends with|| trueand the API pipeline's exit status comes fromhead, so underset -ea failed lookup reaches the 'Could not determine CLI version' error rather than aborting silently.- The diagnostics in both lookup functions go to stderr, so they don't end up in the
tag=$(...)value captured incheck_version.
The new integration-tests/installer_test.go (TestInstallerLatestVersion) runs installer.sh with the raw method against an httptest fake of GitHub and a fake gh. It covers each token source, an invalid token, the 403 and 500 fallbacks, and both lookups failing, and it is skipped on Windows. No test covers a rejected token from an earlier source falling through to a later one, or a network failure during the token check.
Review details
- Commit: 7cc34f7
- Model: claude-opus-5-5
Review 1 of 10 for this pull request · View the full run
A token that GitHub rejected stopped the search, so a valid token from a later source (e.g. ~/.global/auth.json) was not used, and a logged-in gh CLI overrode GITHUB_TOKEN. Now each source is tried in order, with GITHUB_TOKEN first (as with gh itself). Only a 401 counts as a rejection: if the check fails for another reason, the token is kept, and the error is logged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnUBGurqFyfoRBS7aXYqDR
|
📋 PR Summary The installer now finds the latest CLI version from the redirect of Changes
Flow Diagramflowchart TD
A[VERSION pinned?] -->|yes| D[Download]
A -->|no| B[GET releases/latest]
B -->|redirects to tag| D
B -->|no redirect| C[API with each token source in turn]
C -->|401| C
C -->|accepted or other error| E[Use the tag or fail]
C -->|no token accepted| F[API without a token]
F --> E
E --> D
|
Upsun containers, where ~/.global/auth.json is expected, have jq. Without jq, the Composer auth sources give no token, and the other sources are still tried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnUBGurqFyfoRBS7aXYqDR
There was a problem hiding this comment.
Warning
Changes suggested — 🟡 1 warning
🔁 Incremental · 2 files reviewed
Verification
jq -r '."github-oauth"."github.com" | strings'prints nothing, without error, for a missing key, a nullgithub-oauth, or a non-string value. On malformed JSON it fails, and2>/dev/null || trueswallows the error.pathWithoutskips the excluded name and lets an earlier PATH entry win through the Lstat check. The fakeghcomes before it in PATH, so the no-jq case still runs the same installer tools.- Every case that reads Composer auth JSON is marked
needsJQ, so it skips rather than fails where jq is missing. The no-jq case does not depend on jq being installed.
integration-tests/installer_test.go adds a no-jq case and marks the Composer auth cases needsJQ. Those cases skip where jq is not installed, so their coverage depends on the CI runners having jq. Nothing tests the installer against a real Upsun build image.
Review 3 of 10 for this pull request · View the full run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnUBGurqFyfoRBS7aXYqDR
The latest release page's redirect needs no token and is not subject to the API's rate limit, so it is now tried first: a normal install makes one request to find the version, and reads no tokens. If it fails, the API is tried with each token source in order, in the request itself (a 401 means the next source is tried, so the separate /rate_limit check is gone), and then without a token. An error with a token is not retried. A token is sent with the download only if GitHub accepted it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnUBGurqFyfoRBS7aXYqDR
There was a problem hiding this comment.
Warning
Changes suggested — 🟡 1 warning
🔁 Incremental · 2 files reviewed
Verification
latest_tag_from_api_with_authmoves to the next source only while rc=1 (no token, or a 401). On rc=2 it stops without an unauthenticated retry, which matches the "API error with a token is not retried" case.- Output is no longer printed from inside
$(...)command substitutions, so warnings go to the normal stream. This removes the earlier>&2workaround. - When
jqis missing and a token source is still needed, the installer now prints a warning that Composer auth was found but cannot be read, instead of skipping it silently. - The unauthenticated API attempt uses its own
api_errorand the rate-limit hint matchesHTTP 403/HTTP 429, which is theHTTP ${status}format thatlatest_tag_from_apiproduces.
integration-tests/installer_test.go was rewritten. It runs installer.sh against an httptest fake of GitHub and covers the redirect-first order, falling through from each token source to the next, the warning when jq is missing, and the Authorization header sent with downloads. Nothing exercises downloading a pinned version from a URL that needs a token.
Review 5 of 10 for this pull request · View the full run
A URL (e.g. a private mirror) may need the token. GITHUB_TOKEN is now sent with the download unless GitHub rejected it; tokens from the other sources are only sent if GitHub accepted them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnUBGurqFyfoRBS7aXYqDR
Fixes #73.
Unpinned raw installs in Upsun builds failed with "Could not determine CLI version", while
VERSION=5.10.3worked. At the time, v5.10.3 was the latest published release (v5.10.4 was still a draft), so both installs targeted the same release: the only difference was the unpinned install's call to the GitHub API. The likely cause is the API's rate limit for unauthenticated requests (60 per hour per IP), as build hosts share their IPs. The error was hidden (curl -f … 2>/dev/null), so the log did not show it.Changes
https://github.com/upsun/cli/releases/latest(to…/releases/tag/<tag>). It needs no token and is not subject to the API's rate limit, and the download is from github.com anyway. A normal install makes one request to find the version, and reads no tokens.GITHUB_TOKEN(first, as withghitself), theghCLI,COMPOSER_AUTH, then~/.global/auth.json(github-oauth.github.com, read withjq, which Upsun containers have; withoutjqthe installer says so). The last two are where the legacy PHP installer (removed in chore(legacy): remove the legacy PHP installer and self-update #186) found the unprivileged token that project containers provide for higher rate limits. A 401 means the next source is tried; any other error stops there, as another token would not help. If no token is accepted, the API is tried without one, with a hint to setGITHUB_TOKENon a 403 or 429. Each failure is logged.GITHUB_TOKENis sent with the download as before (e.g. for aURLthat needs it), unless GitHub rejected it in the API fallback. Tokens from the other sources are only sent if GitHub accepted them.RELEASES_URLandRELEASES_API_URL(URL, for downloads, now defaults to$RELEASES_URL/download).Tests
integration-tests/installer_test.gorunsinstaller.sh(raw method) against anhttptestfake of GitHub, with an environment built from scratch so that real tokens are not used, and a fakeghCLI. For each case it checks the installed version, the output, and every request: the number of requests to the latest release page (always without a token), theAuthorizationheader of each API request in order, and that of the download.Cases: a pinned version (no lookup,
GITHUB_TOKENsent with the download); the redirect, withGITHUB_TOKENonly sent with the download, and other tokens not used; the API fallback for a 404 and for a page that does not redirect; each token source; source precedence; each rejected source falling through to the next, and to no token (the rejectedGITHUB_TOKENis not sent with the download); Composer auth withoutjq; the API rate limited (403, with the hint), a server error, and an error with a token not being retried. Cases that read Composer auth JSON are skipped ifjqis not installed (it is on the GitHub runners). The tests run in about 1s on Linux and macOS in CI (skipped on Windows). Breaking the order, the fall-through, the no-retry rule, the download token rules, or theauth.jsonsource each fails the relevant cases.Also run against the real GitHub: the redirect, and the API fallback.
The other install methods (apt, yum, apk, Homebrew) still rely on the
test-installerworkflow andscripts/test/installer*.sh, as they need real packages.Not verified: that
~/.global/auth.jsonholds the token in current Upsun builds (it now matters only if the redirect fails).🤖 Generated with Claude Code
https://claude.ai/code/session_01PnUBGurqFyfoRBS7aXYqDR