fix: preserve Anthropic cache_control and cache_creation on /v1/responses

[lukesorvikDO]

Related

• INF-931: cache_control dropped on /v1/responses content blocks (bug 1)
• INF-932: cache creation/read tokens lost in transformation pipeline (bugs 2-4)

Summary

Fixes the 3 LiteLLM-side bugs causing Anthropic prompt caching to silently produce cache_created_input_tokens: 0 / cache_read_input_tokens: 0 on POST /v1/responses. All 3 bugs share the same symptom (zero cache tokens reported) but are independently reproducible.

Rebased onto release/v1.94.0-do.1 (from release/v1.94.0-do.0). Supersedes #38.

Bug 1: cache_control dropped on request content blocks

File: litellm/responses/litellm_completion_transformation/transformation.py

_transform_responses_api_content_to_chat_completion_content rebuilt each content block with only type + text, silently dropping any cache_control field the caller supplied. The downstream Anthropic adapter never sees a caching directive, so the prompt cache is never seeded.

Fix: Copy cache_control from the source item onto the rebuilt content block when present.

---

Bug 2: cache_creation fields dropped in usage transform

File: litellm/responses/litellm_completion_transformation/transformation.py

_transform_chat_completion_usage_to_responses_usage dropped cache_creation_input_tokens, cache_read_input_tokens, and cache_creation.ephemeral_5m/1h_input_tokens when converting internal Usage to ResponseAPIUsage. The inference proxy sees zero cache-creation tokens.

Fix: Use setattr to preserve these as extras on ResponseAPIUsage (which has extra="allow" via BaseLiteLLMOpenAIResponseObject), so they survive serialization and can be extracted by the inference proxy before the lossy api.Usage unmarshal.

---

Bug 3: Wrong variable assignments in streaming usage

File: litellm/llms/anthropic/experimental_pass_through/responses_adapters/streaming_iterator.py

cache_creation_tokens was assigned the entire input_tokens_details object (not a count), and cache_read_tokens was assigned output_tokens_details (completely the wrong field). Streaming responses always reported zero cache tokens.

Fix: Remove the dead/wrong assignments. Read cache_creation_input_tokens and cache_read_input_tokens directly as integers. Add fallback to input_tokens_details.cached_tokens (OpenAI-style field) when the Anthropic-native cache_read_input_tokens is absent.

---

Bug 4 (not in this PR): inference-proxy lossy unmarshal

response_conversions.go ConvertExecutorResponseToAPIResponse does a plain json.Unmarshal of the executor's raw response, which is lossy for these Anthropic-specific extra fields. Even with bugs 1-3 fixed, the inference-proxy needs its own fix (INF-936) to reliably read these fields off the raw executor JSON.

---

Expected behavior after this PR lands (+ INF-936)

Request:

curl -sS https://inference.do-ai.run/v1/responses \
  -H "Authorization: Bearer $ACCESS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic-claude-4.6-sonnet",
    "stream": false,
    "input": [
      {
        "role": "system",
        "content": [
          { "type": "input_text", "text": "<contract text, 2000+ tokens>", "cache_control": { "type": "ephemeral" } }
        ]
      },
      { "role": "user", "content": "What are the key terms?" }
    ]
  }'

Before (bug):

"usage": {
  "input_tokens": 1936, "output_tokens": 246, "total_tokens": 2182,
  "cache_created_input_tokens": 0, "cache_read_input_tokens": 0, "cache_creation": null
}

After (first request — cache write):

"usage": {
  "input_tokens": 1936, "output_tokens": 246, "total_tokens": 2182,
  "cache_created_input_tokens": 1900, "cache_read_input_tokens": 0,
  "cache_creation": { "ephemeral_5m_input_tokens": 1900, "ephemeral_1h_input_tokens": 0 }
}

After (follow-up identical request — cache read):

"usage": {
  "input_tokens": 1936, "output_tokens": 246, "total_tokens": 2182,
  "cache_created_input_tokens": 0, "cache_read_input_tokens": 1900,
  "cache_creation": { "ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0 }
}

Sequencing

• Must land alongside/after INF-931 for cache activity to show up in billing.
• Needs INF-936's inference-proxy fix for values to survive the response conversion.
• Blocks INF-934 and INF-935.

Pre-Submission checklist

• My PR's scope is as isolated as possible; it only solves 1 specific problem
• Tests added for all 3 bugs

Type

🐛 Bug Fix

Made with Cursor

[lukesorvikDO]

outdated