review: clamp the run budget to the effective credit cap; land before the cap#249
review: clamp the run budget to the effective credit cap; land before the cap#249jwbron wants to merge 13 commits into
Conversation
🦋 Changeset detectedLatest commit: d87f728 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
khan-actions-bot
requested review from
a team,
kevinb-khan and
somewhatabstract
and removed request for
a team
…ve credit cap; land before the cap The tier budget table assumes the workflow's $10 ceiling, but the per-run credit cap is enforced runner-side and was invisible to the agent, so a tighter consumer cap produced soft targets the hard cap could not pay for: the 400-credit budget-shed behavior test planned against the high tier's $10 targets, spent 390 credits on the finder phase, and was killed by the api-proxy mid-persist with findings in hand and nothing posted. Three changes close the gap: - The frontmatter sets max-ai-credits: 1000 explicitly and mirrors it into the agent environment as REVIEW_MAX_AI_CREDITS (consumers that override the cap keep the two in sync). - The router resolves the effective cap (mirror, GH_AW_MAX_AI_CREDITS, a visible awf config, then the 1000-credit gh-aw default) and clamps the selected tier budget proportionally, floored at the trivial tier's values; routing.json carries capClamped and effectiveCreditCap so the orchestrator and post-hoc audits see the clamp. - The budget prompt adds an estimated-credits proxy (in-band subagent_tokens sum / 5000) and two mandatory checkpoints, the critical one after the last finder returns and before claim validation, the exact boundary the observed run died at.
jwbron
force-pushed
the
jwies/review-budget-cap
branch
from
4370d5b to
78e501e
Compare
jwbron
changed the base branch from
jwbron/review-out-artifact-upload
to
jwbron/review-skill-rule-quote
|
Restacked for a linear chain: this PR now bases on jwbron/review-skill-rule-quote (#248), so the full order is #243 -> #244 -> #245 -> #247 -> #248 -> #249 -> #250, with #246 (mode dial) the one sibling off #247 while its eval updates land. Content unchanged (cherry-picks 78e501e / 45cd748); reset any local state to origin. Acceptance re-run mechanics for this PR (the budget-shed behavior test, per the Khan/webapp#40737 record): the old scaffold base branch (review-eval/budget-shed-base) was deleted, but webapp's review-bot-v1.4.0-preview branch is still alive. Recreate: branch off review-bot-v1.4.0-preview, re-pin review-preview.md's pre-agent-steps ref and source: to jwies/review-budget-cap (branch ref), set max-ai-credits 400, gh aw compile, push; stack a copy of the seeded PR on it and confirm the run lands a verdict with skipped-dimension notes instead of dying at the cap. |
router.ts crossed the 1000-line lint budget with the clamp addition; the cap discovery (resolveCreditCap), the clamp (clampBudgetToCreditCap), and the gh-aw default constant move to their own module by concern. The router imports both and stays the single CLI entry point; tests for the moved functions import from the new module. Prettier fix in router.test.ts rides along.
Review live A/BBaseline:
Adversarial hard gate: PASSED on the candidate arm. Agent failures
Single-run-stable rows: recall, verdict agreement, regressions, adversarial gate. Judge quality and noise are not: they jitter run-to-run at this corpus size, and a regressed reviewer can score HIGHER on judge quality (fewer, surer comments each read better). Recall against the labeled specs is the load-bearing metric. |
Behavior-test record: budget shed re-run (jwies/review-budget-cap pin, max-ai-credits 400)Run: 29056727919, agent job failure, ~8m agent runtime. Cost: 394.5 of 400 credits (proxy denial at 416.8/400 in-flight). No review posted. Outcome vs pass criteria: FAIL, but the failure moved one layer up.
Scaffold ( |
…edit cap The acceptance re-run showed the clamp engaging and the run still dying at the hard cap (416/400 credits, mid-skill-audit): the clamped soft target equaled the cap, spend is unobservable mid-run, and requests in flight at the last checkpoint bill after it. The clamp now sets the soft dollar target to 75% of the effective cap (LANDING_RESERVE_RATIO) and scales the other targets to match, so shedding against the targets lands the run inside the ceiling. The clamp also engages whenever the tier's own target sits above the reserve-adjusted cap, not only above the raw cap. review.md tells the orchestrator maxUsd is a landing target, never money it may finish spending. Acceptance: re-run the budget-shed test on the kept-alive webapp scaffold at cap 400.
…ies/review-budget-cap
…fetch signature (eslint no-undef)
…to jwbron/live-eval-ab-ci
…o jwbron/review-trial-skill
…-skill' into jwbron/review-out-artifact-upload
…ut-artifact-upload' into jwbron/review-rereview-accountability
…untability' into jwbron/review-out-of-lane
… into jwbron/review-dispatch-tax
… tmp/skill-rule-quote
…ies/review-budget-cap
1 participant
Closes the budget-shed gap the v1.4.0-preview behavior test exposed (Khan/webapp#40737, run 29051903691): the tier budget table is sized inside the workflow's assumed $10 ceiling, but the per-run credit cap is enforced runner-side by the firewall api-proxy and is invisible in-container. A consumer with a tighter
max-ai-creditstherefore got soft targets the hard cap could not pay for: the 400-credit test run planned against the high tier's $10 targets (its own words: the cap was 'unobservable to me mid-run'), shed only the opt-in reviewers, spent 390 credits on the finder phase, and was killed by the api-proxy mid-persist with findings in hand and nothing posted.Changes
max-ai-credits: 1000is now explicit (matches the gh-aw default) and mirrored into the agent environment asenv: REVIEW_MAX_AI_CREDITS, with a keep-in-sync note for consumer overrides. gh-aw compilesenv:to workflow-level env and the firewall's--env-allforwards it into the agent container (verified against a scratch compile and the v0.81.6 lock).lib/router.ts):resolveCreditCapresolves the effective cap (mirror env,GH_AW_MAX_AI_CREDITS, a visible awf config'sapiProxy.maxAiCredits, then the 1000-credit default);clampBudgetToCreditCapscales the selected tier's soft targets proportionally, floored at the trivial tier's values, and marksrunBudget.capClamped+runBudget.effectiveCreditCapin routing.json. Zero/negative caps mean explicitly uncapped; the pureroute()core takes the cap as config, so non-CLI callers (eval suite) are unaffected.capClampedmeans dispatch-conservatively; adds an estimated-credits proxy (sum of in-bandsubagent_tokensover completed sub-agents divided by 5,000, derivation documented inline from measured runs at ~9,000 summed tokens per credit); makes two proxy checkpoints mandatory, the critical one immediately after the last finder returns and before Phase 3 validation, the exact boundary the observed run died at.With the clamp, the 400-credit scenario's targets become 4 invocations / 8 minutes / $4, so the existing three-quarters rule trips during the finder phase and the run sheds stages 2-3 and lands a partial review instead of dying.
Test plan
pnpm run test --run: 708 tests green (new: clamp scaling incl. the 400-credit incident shape, trivial floors, uncapped sentinels, resolveCreditCap source priority and fallbacks, runCli env integration).pnpm run typecheck: clean.Stacked on #243 (batch two).
Human steps (non-review)
review-bot-v1.4.0-preview(still alive), re-pinreview-preview.md'spre-agent-stepsref andsource:tojwies/review-budget-cap, setmax-ai-credits: 400,gh aw compile, push, and stack a copy of the seeded PR on it. Pass = the run lands a verdict with skipped-dimension notes instead of dying at the cap. ~$4 of credits.Update 2026-07-09: landing reserve
The acceptance re-run showed the clamp engaging and the run still dying at the hard cap (416/400 mid-skill-audit): the clamped soft target equaled the cap, and spend that is unobservable mid-run plus in-flight work billing after the last checkpoint left no room to land. The clamp now sets the soft dollar target to 75% of the effective cap (
LANDING_RESERVE_RATIO) and scales the other targets to match, engages whenever the tier target sits above the reserve-adjusted cap (not only above the raw cap), and review.md now tells the orchestrator thatmaxUsdis a landing target, never money it may finish spending. Acceptance: re-run the budget-shed test on the kept-alive webapp scaffold at cap 400; expected shape is shed-early, land inside 400.