fix: map OpenAI cached_tokens to cache_read_input_tokens on /v1/messages - #37
Open
lukesorvikDO wants to merge 1 commit into
Conversation
translate_response in LiteLLMAnthropicToResponsesAPIAdapter only copied input_tokens and output_tokens from the OpenAI Responses API usage, dropping input_tokens_details.cached_tokens. The inference-proxy always saw cache_read_input_tokens=0 even when OpenAI returned a cache hit. Fix: map input_tokens_details.cached_tokens to cache_read_input_tokens in both the non-streaming translate_response and the streaming iterator. Co-authored-by: Cursor <cursoragent@cursor.com>
2 tasks
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
translate_responseinLiteLLMAnthropicToResponsesAPIAdapteronly copiedinput_tokensandoutput_tokensfrom the OpenAI Responses API usage, droppinginput_tokens_details.cached_tokens. As a result, inference-proxy always sawcache_read_input_tokens=0on/v1/messageseven when OpenAI returned a cache hit.Fix
Map
input_tokens_details.cached_tokenstocache_read_input_tokensin both the non-streamingtranslate_responsepath and the streaming iterator (AnthropicResponsesStreamWrapper), falling back to the OpenAI-style field only when the Anthropic-stylecache_read_input_tokensfield isn't already present.Scope
This PR is split out of #35 to keep each fix atomic. It contains only the OpenAI
/v1/messagescache-read mapping fix. The Anthropic/v1/responsescache_controlandcache_creationfixes from #35 are tracked separately.Pre-Submission checklist
Type
🐛 Bug Fix
Made with Cursor