Skip to content

Return 405 on GET /mcp so idle SSE streams stop exhausting Cloud Run - #4181

Merged
matthewhelmke merged 1 commit into
mainfrom
exp-577-mcp-refuse-sse-get
Oct 9, 2026
Merged

matthewhelmke merged 1 commit into
mainfrom
exp-577-mcp-refuse-sse-get

Conversation

@matthewhelmke

Copy link
Copy Markdown
Collaborator

The hosted docs MCP server returned 429 Rate exceeded to a customer's MCP gateway, and to every other client using the server at the time (EXP-577). The cause was server-to-client SSE streams that the server accepts but never uses. This PR makes the server refuse them with 405, which the MCP spec defines for exactly this case.

Cause

  • The 429 isn't a configured rate limit. It's Cloud Run's response when no instance has a free request slot. The service allows 3 instances at Terraform's default concurrency of 80, so 240 requests can be in flight at once, shared by every client.
  • Since fix(mcp): run docs MCP server stateless to fix cross-instance 404s (CUS-1340) #4055 the server runs stateless. In that mode the MCP Python SDK still accepts GET /mcp from clients on the 2025 protocol versions. It holds the stream open with a keepalive ping every 15 seconds and never sends anything, because a stateless server has nothing to send.
  • Cloud Run closes each stream at its 300-second request timeout, so every GET holds a slot for five minutes. Measured against the live server: 302.6 seconds.
  • In the customer's report, 286 GETs arrived in five minutes. At five minutes each, that's about 286 open streams against 240 slots. Once the slots fill, Cloud Run returns 429 to everything, including initialize.

This is an upstream SDK bug: modelcontextprotocol/python-sdk#2474, still open, with no fix in 2.3.0.

Changes

scripts/mcp-server.py

  • A new build_http_app() builds the SDK's Starlette app with the same stateless_http=True and json_response=True settings as before. It then inserts a GET-only route ahead of the SDK's /mcp route, and that route returns 405 with Allow: POST.
  • Starlette matches routes in order and the new route accepts only GET and HEAD, so POST and DELETE still reach the SDK. The SDK's custom_route() can't do this, because it adds routes after /mcp.
  • The HTTP branch runs that app with uvicorn.run() instead of server.run(). It passes the same host, port, and log level the SDK's runner passes, and uvicorn still reads FORWARDED_ALLOW_IPS from the environment, so the fix(mcp): keep HTTP redirects on https behind Cloud Run (DOCS-99 root cause) #3704 redirect fix is unaffected.
  • The CUS-1340 explanation moves into the new function's docstring, which also covers EXP-577. Stdio mode is unchanged.

scripts/test_mcp_transport.py

  • The source-inspection test, which looked for a server.run(... stateless_http=True) call, is replaced by functional tests that send requests to the app production serves:
    • GET returns 405 with Allow: POST, with and without a protocol-version header.
    • initialize returns a JSON result and mints no Mcp-Session-Id.
    • tools/list works without a session.
  • Starlette's TestClient runs as a context manager so the app's lifespan starts the SDK's session manager. Without it, every POST returns 500.

.github/workflows/mcp-image-smoke-test.yaml

  • The unit-test job installs pytest-timeout. If the 405 route regresses, the GET opens a stream that never ends, and the test hangs instead of failing. The GET test has a 10-second timeout, and the job has no timeout-minutes of its own.

scripts/smoke-test-mcp.sh

  • New Check 5 sends a GET to the built image and expects 405. The comment that explained why the script never sends GET is updated.

Testing

  • pytest scripts/test_mcp_server.py scripts/test_mcp_transport.py scripts/test_compile_docs.py: 48 passed.
  • Each new test fails when its setting is broken:
    • Without the 405 route, the GET tests hang, and pytest-timeout fails them after 10 seconds.
    • With stateless_http=False, the initialize and tools/list tests fail.
    • With json_response=False, the same two tests fail.
  • Built the image from this branch and from main with CI's placeholder artifacts. smoke-test-mcp.sh passes all five checks on this branch. On main, Check 5 fails with a 200.
  • Ran two real clients against the container:
    • Claude Code, which uses the MCP TypeScript SDK, sent one GET after initialize, got the 405, didn't retry, and completed a search_docs call.
    • The MCP Python SDK client lists all five tools and calls search_docs. It never sends a GET.
  • POST /mcp/ with X-Forwarded-Proto: https still redirects to https://mcp.edu.chainguard.dev/mcp.

A code review of the Envoy AI Gateway and the MCP TypeScript SDK found that both treat a 405 on GET as "no stream" and don't retry. Cursor's own client, Goose, and older TypeScript SDK versions weren't checked.

Not in this PR

  • No infrastructure change. Without the idle streams, 240 slots is plenty for short tool calls. If 429s continue after the deploy, the next step is a shorter Cloud Run request timeout in iac/.
  • I haven't seen server logs or metrics, because Docs has no access to the GCP project. A log query for httpRequest.status=429 on the mcp-server service for 2026-09-30, 16:17 to 16:22 UTC would confirm where the 429s came from.

After merge

The scripts/** push trigger redeploys the hosted server. Then a GET to https://mcp.edu.chainguard.dev/mcp with MCP-Protocol-Version: 2025-11-25 should return 405 right away instead of holding the stream open.

Refs EXP-577, CUS-1340

🤖 Generated with Claude Code


Created in collaboration with Claude Code running Opus 5.5 on 2026-10-09.

In stateless mode the MCP SDK still accepts GET /mcp and holds an empty
SSE stream open until Cloud Run's 300-second request timeout. Each
stream holds one of the service's 240 request slots (3 instances x 80),
and a customer's MCP gateway opened enough of them that Cloud Run
returned 429 to every caller (EXP-577). Upstream issue:
modelcontextprotocol/python-sdk#2474.

The server now builds its Starlette app itself, puts a GET-only route
that returns 405 (Allow: POST) ahead of the SDK's /mcp route, and runs
it with uvicorn. POST and DELETE still reach the SDK.

The source-inspection transport test becomes a functional test of the
production app, with pytest-timeout so a regression fails instead of
hanging. The smoke test gains a GET check against the built image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@matthewhelmke
matthewhelmke requested review from a team as code owners October 9, 2026 17:49
@vercel

vercel Bot commented Oct 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
chainguard-docs-preview Ready Ready Preview Oct 9, 2026 5:49pm UTC

Request Review

@netlify

netlify Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for ornate-narwhal-088216 ready!

Name Link
🔨 Latest commit bb2b930
🔍 Latest deploy log https://app.netlify.com/projects/ornate-narwhal-088216/deploys/6ac9291db681f00008dfde6e
😎 Deploy Preview https://deploy-preview-4181--ornate-narwhal-088216.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@matthewhelmke
matthewhelmke merged commit f2f104d into main Oct 9, 2026
18 checks passed
@matthewhelmke
matthewhelmke deleted the exp-577-mcp-refuse-sse-get branch October 9, 2026 18:03

This branch was successfully deployed

1 active deployment
Preview — bb2b9307 Deployed Oct 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant