fix(server): surface Pool capacity exhaustion

[hpliStartAgain]

Summary

• Publish a PoolAllocationPending BatchSandbox condition when fixed or automatically selected Pools cannot satisfy the remaining allocation because they are at poolMax.
• Let the lifecycle Server wait for a bounded Pool acquisition window, then return HTTP 429 with KUBERNETES::POOL_CAPACITY_EXHAUSTED and Retry-After instead of the generic Pod readiness timeout.
• Keep the existing create rollback path, OpenAPI contract, generated SDKs, configuration reference, and Kubernetes operations documentation aligned.

Fixes #1578.

Problem

A create request using extensions.poolRef can remain pending for the entire sandbox create timeout when the selected Pool has reached poolMax. The Server currently cannot distinguish this allocation back-pressure from an ordinary Pending workload, so the request eventually returns 504 KUBERNETES::POD_READY_TIMEOUT.

Pool capacity pressure and Pod startup failures require different caller and operator actions. Capacity may recover while the request is waiting, but persistent exhaustion should remain machine-identifiable.

Minimal reproduction

1. Create a compatible Pool with poolMax: 2.
2. Allocate two sandboxes from that Pool and confirm 2 allocated / 0 available.
3. Submit a third POST /v1/sandboxes request with the same extensions.poolRef.
4. Keep the Pool at capacity.

Before this change, the request waits for the overall create timeout (approximately 180 seconds with the reproduced configuration) and returns KUBERNETES::POD_READY_TIMEOUT.

Root cause

The controller enforces capacity, but neither allocation path exposed a stable status that the lifecycle Server could consume:

• a fixed poolRef remained Pending while waiting for a Pool Pod;
• poolRef: "*" returned an untyped no-eligible-Pool error and recorded a Kubernetes Event;
• _wait_for_sandbox_ready mapped every remaining Pending state to the generic readiness timeout.

A Server-side preflight check would be racy because concurrent creates can consume capacity after the check and before controller allocation.

Change

• Add PoolAllocationPending with reason PoolCapacityExhausted to the BatchSandbox status contract and generated/Helm CRDs.
• Preserve stable predicate failure codes for automatic Pool assignment, and classify capacity only when a Pool passed every other predicate.
• Publish and clear the capacity condition for both fixed and automatic Pool selection; requeue while an allocation is waiting so released capacity can be consumed.
• Add kubernetes.pool_acquisition_timeout_seconds (default: 30 seconds). It measures continuous capacity blockage and never extends sandbox_create_timeout_seconds.
• Map persistent capacity exhaustion to HTTP 429, KUBERNETES::POOL_CAPACITY_EXHAUSTED, and Retry-After: 5.
• Reuse the existing create rollback path so a timed-out BatchSandbox is deleted.
• Document the behavior and regenerate the affected Python and JavaScript lifecycle clients.

The HTTP status is intentionally isolated from the machine-readable error code. I used 429 to represent Pool admission back-pressure, but can switch to 503 if that better matches the project's API convention.

Testing

• Not run (the changed controller/server images were not deployed to a real cluster; make helm-lint was also unavailable because Helm is not installed locally)
• Unit tests
• Integration tests
• e2e / manual verification

Completed locally:

• cd kubernetes && make test
• cd kubernetes && make lint
• cd kubernetes && make build
• focused controller tests for fixed Pool exhaustion, automatic Pool exhaustion, capacity release, non-capacity predicate failures, and inactive lifecycle phases
• cd server && uv run ruff check
• cd server && uv run pytest (1484 passed)
• Python SDK generation, Ruff, Pyright, and full tests
• JavaScript SDK generation, lint, typecheck, build, and 94 Node tests
• Kotlin lifecycle generation, Spotless, sandbox tests, and code-interpreter tests
• cd docs && pnpm docs:build
• license, diff, generated CRD alignment, and sensitive-information checks

The reproduced fixed-Pool behavior used Server 0.2.2 and Controller 0.2.0. A real-cluster run with the modified images remains outstanding.

Breaking Changes

• None
• Yes (describe impact and migration path)

This is additive: a new CRD condition, configuration field, documented 429 response, and generated response handling. A new Server with an older controller safely falls back to the existing Pending/504 behavior because the condition is absent; an older Server ignores the new condition. Pool-not-found, genuine unschedulable, image failure, and ordinary Pod readiness timeout classifications remain unchanged.

Checklist

• Linked Issue or clearly described motivation
• Added/updated docs (if needed)
• Added/updated tests (if needed)
• Security impact considered
• Backward compatibility considered

No authentication or authorization boundary changes. The 429 response does not expose credentials, tenant data, Pod names, or internal infrastructure details, and the generated SDKs do not automatically replay the non-idempotent create request.

[Pangjiping]

Reviewed the Pool capacity exhaustion change. Overall the approach is solid (conservative capacity classification, additive condition, consistent spec/SDK/CRD/docs updates). A few inline comments below, mostly around requeue gating, condition lifecycle, and status-patch consistency.

[hpliStartAgain]

Hi @Pangjiping, thank you for approving the PR and updating the branch with main.

The refreshed CI is currently blocked by Java E2E (docker bridge), which fails during test compilation because SandboxPool.Builder.reconcileInterval(Duration) cannot be resolved. This PR does not modify the Java SDK or Java E2E files, and the Java E2E passed on the same base commit, so the failure appears unrelated to the PR diff.

I do not have permission to rerun Actions in the upstream repository. Could you please rerun the failed jobs when convenient? Thanks!