Add file frames and universal LLM context support for files by mattieruth · Pull Request #5826 · pipecat-ai/pipecat

mattieruth · 2026-09-17T19:41:11Z

Part 1 of File Support Work
Part 1
Part 2
Part 3
Part 4

Corresponding Client-JS changes
New File Input Example

Introduce FileRawFrame, InputFileRawFrame, and UserFileRawFrame, carrying a file payload as a base64 data URL or a URL. LLMContext gains create_file_message(), create_file_url_message(), and add_file_frame_message(), producing universal file_base64 and file_url content items that each adapter (Anthropic, AWS Bedrock, Gemini, OpenAI Chat, OpenAI Responses) translates to its provider format, mirroring the existing image support. The assistant aggregator appends UserFileRawFrame to the context, with run_llm controlling whether it triggers a completion.

Unsupported MIME types or source combinations raise during conversion rather than being silently dropped, and
LLMContext.remove_invalid_file_message() lets the OpenAI, Anthropic, Bedrock, and Gemini services drop the offending file message when a request is rejected, so the context can't get permanently stuck.

The minimum aiohttp version is now 3.14.0 (required by google-genai >= 2.8.0).

codecov · 2026-09-17T19:48:53Z

Codecov Report

❌ Patch coverage is 85.12397% with 36 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/pipecat/adapters/services/anthropic_adapter.py	64.70%	12 Missing ⚠️
src/pipecat/adapters/services/bedrock_adapter.py	80.64%	12 Missing ⚠️
src/pipecat/adapters/services/gemini_adapter.py	50.00%	4 Missing ⚠️
...cat/adapters/services/open_ai_responses_adapter.py	75.00%	2 Missing ⚠️
src/pipecat/frames/frames.py	91.30%	2 Missing ⚠️
src/pipecat/processors/aggregators/llm_context.py	95.34%	2 Missing ⚠️
src/pipecat/adapters/services/open_ai_adapter.py	95.00%	1 Missing ⚠️
...t/processors/aggregators/llm_response_universal.py	95.23%	1 Missing ⚠️

Files with missing lines	Coverage Δ
src/pipecat/services/anthropic/llm.py	`79.84% <100.00%> (+2.01%)`	⬆️
src/pipecat/services/aws/llm.py	`71.27% <100.00%> (+2.07%)`	⬆️
src/pipecat/services/google/llm.py	`75.15% <100.00%> (+1.53%)`	⬆️
src/pipecat/services/openai/base_llm.py	`78.41% <100.00%> (+0.58%)`	⬆️
src/pipecat/adapters/services/open_ai_adapter.py	`97.46% <95.00%> (-0.90%)`	⬇️
...t/processors/aggregators/llm_response_universal.py	`92.30% <95.23%> (+1.22%)`	⬆️
...cat/adapters/services/open_ai_responses_adapter.py	`94.82% <75.00%> (-1.47%)`	⬇️
src/pipecat/frames/frames.py	`95.90% <91.30%> (+0.13%)`	⬆️
src/pipecat/processors/aggregators/llm_context.py	`91.19% <95.34%> (+12.24%)`	⬆️
src/pipecat/adapters/services/gemini_adapter.py	`70.51% <50.00%> (-0.52%)`	⬇️
... and 2 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

filipi87 · 2026-09-28T13:53:49Z

 @dataclass
 class UserImageRawFrame(InputImageRawFrame):


Would it make sense for UserImageRawFrame to extend UserFileRawFrame? Or are we still handling them differently enough?

maybe? I think originally I was attempting to not touch the image stuff. i'll look at it.

ok i/claude took a look and while i agree they feel like they can/should be combined, here's the gotcha:

ImageRawFrame and FileRawFrame are genuinely different. ImageRawFrame is raw bytes + size, while FileRawFrame is an opaque string that encompasses either a URL or base64 data URL.

Claude gave a lengthier answer with more reasoning that i'm not sure i agree with, but i think the big one is the core difference of what is stored. And since ImageRawFrame is pre-existing, re-molding it to work as a File is even trickier.

I think this is okay for now. Let's save this refactor until all the file support lands, and only do it if we feel we need it.

filipi87 · 2026-09-28T14:03:36Z

+        if image_appended and frame.run_llm is not False:
+            await self.push_context_frame(FrameDirection.UPSTREAM)
+
+    async def _handle_user_file_frame(self, frame: UserFileRawFrame):


Claude mentioned this possible issue here:

Medium — src/pipecat/processors/aggregators/llm_response_universal.py:2090

_handle_user_file_frame adds the file message to the context before it calls push_aggregation(). If the assistant aggregator is holding text (the bot is partway through a reply when the file arrives), that partial reply is written as an assistant message after the user's file message, so the context order is wrong.

This also defeats the cleanup. LLMContext.remove_invalid_file_message() only looks at messages after the last assistant message, so a file that sits before that misplaced assistant message is never removed. If the provider rejects the file, every later completion fails on it — the stuck state the cleanup is meant to prevent.

The direct push_aggregation() call also skips _trigger_assistant_turn_stopped, so on_assistant_turn_stopped and LLMContextAssistantTurnFrame are never emitted for the text it commits. The image path (_handle_user_image_frame) does not flush the aggregation at all.

Suggested fix: flush before appending the file, preferably with await self._trigger_assistant_turn_stopped() rather than a bare push_aggregation(). Otherwise, drop the flush to match the image handler.

fixed push_aggregation to now get called before the file load. note: this ordering was also in the image handling, so claude went ahead and fixed it there too. Claude wanted me to tell you this, though, about the _trigger_assistant_turn_stopped() comment and why it refused to make that change 😂

Note on the _trigger_assistant_turn_stopped part of the comment: I deliberately did not close the turn there — closing it mid-stream would make the turn-stopped guard swallow the rest of the reply at LLMFullResponseEndFrame. The plain flush keeps the turn open (remaining text commits at the natural turn end as a second assistant message) and turn events fire once, at the real boundary. Worth saying that in your reply so the "skips turn-stopped" half of the comment is addressed explicitly. New test pins the [assistant, user-file] ordering with in-progress aggregation.

plain flush keeps the turn open (remaining text commits at the natural turn end as a second assistant message) and turn events fire once, at the real boundary

I think this makes sense. I’m not even sure if it’s possible for us to hit this race condition, where a file is sent while the LLM is replying. But in any case, what we have now feels like enough.

I added some comments below. I believe even invoking push_aggregation directly here maybe isn’t the right approach, since we only want to add the messages to the context.

filipi87 · 2026-09-28T14:05:38Z

            except Exception as e:
                await self.push_error(error_msg=f"Error during completion: {e}", exception=e)
+                # OpenAI explicitly flags the invalid file_id parameter.
+                if (
+                    getattr(e, "type", None) == "invalid_request_error"
+                    and getattr(e, "param", None) == "file_id"
+                ):
+                    frame.context.remove_invalid_file_message()


Claude also found another possible issue here:

Medium — src/pipecat/services/openai/base_llm.py:626

The OpenAI service only removes a file after an API error when e.type == "invalid_request_error" and e.param == "file_id". The OpenAI adapter never sends a file_id. It only sends inline {"type": "file", "file": {"filename", "file_data"}} blocks, so when OpenAI rejects a corrupt, encrypted, or oversized PDF, param is the file_data path or null. The cleanup never runs, and the bad file is resent and fails on every later turn. Unsupported MIME types and file URLs are still cleaned up, because they fail earlier with LLMContextConversionError.

Suggested fix: use the same rule as the Anthropic service. Remove the file on openai.BadRequestError, UnprocessableEntityError, or HTTP 413, or at least also match a param that refers to the file content (e.g. one containing file_data).

good catch. fixed.

filipi87 · 2026-09-29T15:40:32Z

+            # Commit any in-progress assistant aggregation before appending,
+            # so the image message lands after it in the context.
+            await self.push_aggregation()


I believe we may need some variation of this where we only add the message to the context.

Because, if I’m not mistaken, push_aggregation will always invoke push_context_frame, which I don’t think is what we want in this case.

The same applies to _handle_user_file_frame.

filipi87 · 2026-09-29T15:41:53Z

+        # message written before a later-committed assistant message could
+        # never be cleaned up if the provider rejects it. The assistant turn
+        # stays open; any remaining reply text commits at the turn's end.
+        await self.push_aggregation()


Same here. We only want to add the in progress assistant aggregation to the context, but not perform the other things that self.push_aggregation does, like invoking self.push_context_frame

filipi87

The only thing missing is the current self.push_aggregation that we’re doing inside _handle_user_image_frame and _handle_user_file_frame. Maybe leave that commented out for now, and we can handle it in the Part 3 PR.

As I mentioned, I believe we may need some variation of this where we only add the message to the context.

Because, if I’m not mistaken, push_aggregation will always invoke push_context_frame, which I don’t think is what we want in this case.

But I’m already leaving my approval here so we can unblock this, merge Parts 1 and 2, and start reviewing Part 3. 🔥

Introduce FileRawFrame, InputFileRawFrame, and UserFileRawFrame, carrying a file payload as a base64 data URL or a URL. LLMContext gains create_file_message(), create_file_url_message(), and add_file_frame_message(), producing universal file_base64 and file_url content items that each adapter (Anthropic, AWS Bedrock, Gemini, OpenAI Chat, OpenAI Responses) translates to its provider format, mirroring the existing image support. The assistant aggregator appends UserFileRawFrame to the context, with run_llm controlling whether it triggers a completion. Unsupported MIME types or source combinations raise during conversion rather than being silently dropped, and LLMContext.remove_invalid_file_message() lets the OpenAI, Anthropic, Bedrock, and Gemini services drop the offending file message when a request is rejected, so the context can't get permanently stuck. The minimum aiohttp version is now 3.14.0 (required by google-genai >= 2.8.0).

1. openai: clean up rejected file messages on payload-shaped API errors The cleanup previously matched an invalid_request_error with param == "file_id", which never occurs: the OpenAI adapter sends files inline as file_data, so a rejected file surfaces as a plain bad-request error and the invalid file message was resent, and failed, on every later turn. Match error classes instead, the same rule as the Anthropic service: bad request (400), unprocessable content (422), and payload too large (413, matched by status code as OpenAI has no dedicated exception class for it) remove the pending file message; the rest of the 4xx range says nothing about whether the file was bad, so it's kept for retry. The tests exercised the old condition with RuntimeErrors carrying bolted-on type/param attributes — which is how they passed while the production check was unreachable — and now build real OpenAI exceptions. 2. aggregator: commit in-progress assistant text before appending a user file A file (or image) arriving while the assistant aggregator holds partial reply text now flushes that aggregation before the user message is appended, so the context keeps chronological order. The order matters beyond chronology: LLMContext.remove_invalid_file_message() only considers messages after the last assistant message, so a file message written before a later-committed assistant message could never be cleaned up when the provider rejects it. The flush is push_aggregation() rather than closing the assistant turn: the turn stays open, any remaining reply text commits at the turn's natural end as a second assistant message, and the turn start/stop events still fire once at the real boundaries.

Committing in-progress assistant text ahead of an arriving user file or image now goes through _add_aggregation_to_context, which writes the context message and nothing else. push_aggregation (unchanged for its existing callers) additionally pushes the downstream LLMContextFrame and assistant-timestamp frame — turn-boundary signals that belong to the turn's real end, not to a mid-turn flush. Routing the flush through it also cleared the push-context-on-bot-stopped-speaking flag as a side effect, which could swallow a pending context push.

mattieruth added this pull request to stack #5828 September 17, 2026 19:43

mattieruth mentioned this pull request Sep 17, 2026

rtvi: add send-file client message #5827

Merged

This was referenced Sep 17, 2026

Add runner file uploads with pluggable storage, resolved at the LLM service #5829

Merged

runner: session-scoped upload route with session-end cleanup #5830

Merged

mattieruth requested review from aconchillo, filipi87, kompfner and markbackman September 17, 2026 20:11

filipi87 reviewed Sep 28, 2026

View reviewed changes

mattieruth force-pushed the mrkb/file-support-part-1 branch from 78838e6 to f281181 Compare September 29, 2026 14:25

filipi87 reviewed Sep 29, 2026

View reviewed changes

filipi87 approved these changes Oct 1, 2026

View reviewed changes

mattieruth force-pushed the mrkb/file-support-part-1 branch 3 times, most recently from 64e0af3 to e1fc821 Compare October 6, 2026 13:57

kompfner approved these changes Oct 6, 2026

View reviewed changes

mattieruth added 3 commits October 7, 2026 10:07

mattieruth force-pushed the mrkb/file-support-part-1 branch from e1fc821 to 476a61e Compare October 7, 2026 14:11

mattieruth merged commit 7d15636 into main Oct 7, 2026
11 checks passed

mattieruth deleted the mrkb/file-support-part-1 branch October 7, 2026 15:25

pipecat-docs-bot Bot mentioned this pull request Oct 7, 2026

docs: update for pipecat PR #5826 pipecat-ai/docs#1440

Merged

mattieruth restored the mrkb/file-support-part-1 branch October 7, 2026 15:59

mattieruth deleted the mrkb/file-support-part-1 branch October 7, 2026 19:23

Conversation

mattieruth commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

filipi87 Sep 28, 2026 • edited by mattieruth Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

filipi87 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mattieruth commented Sep 17, 2026 •

edited

Loading

codecov Bot commented Sep 17, 2026 •

edited

Loading

filipi87 Sep 28, 2026 •

edited by mattieruth

Loading