Skip to content

Commit 96b0ada

Browse files
authored
Merge pull request #1919 from MohammadHijjawi97/fix-detect-truncated-responses
Surface output-limit truncation instead of misformat retries
2 parents ff2a5b9 + 675e7d0 commit 96b0ada

18 files changed

Lines changed: 306 additions & 10 deletions

‎extensions/python/_functions/agent/Agent/hist_add_warning/end/_90_stop_unusable_response_loop.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ def execute(self, data: dict | None = None, **kwargs):
2525
self.agent.read_prompt("fw.msg_empty_response.md"),
2626
self.agent.read_prompt("fw.msg_reasoning_only.md"),
2727
self.agent.read_prompt("fw.msg_thoughts_fallback.md"),
28+
self.agent.read_prompt("fw.msg_output_limit.md"),
2829
}:
2930
return
3031

‎extensions/python/message_loop_result/AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
## Ownership
88

99
- Extensions receive mutable `result_data` with `llm_result` and may set `skip_default_processing` after fully handling the turn.
10+
- `_05_truncated_response.py` retries turns the provider cut off at the output token limit (`LLMResult.truncated`) that carry no complete tool request: add an agent-facing `fw.msg_output_limit.md` history warning with a separate `fw.msg_output_limit_response.md` user notice naming the finish reason, instead of the generic misformat warning, and count them through the stop-unusable-response-loop extension. Native calls are complete only when every raw argument value is a JSON object; a synthesized envelope around malformed arguments does not qualify.
1011
- `_20_empty_response.py` retries fully empty turns (no response and no reasoning): count them toward the unusable-response limit without adding a warning to model history, and use `fw.msg_empty_response.md` for agent-prefixed UI warning text only.
1112
- `_20_empty_response.py` retries reasoning-only turns: add an agent-facing `fw.msg_reasoning_only.md` history warning with a separate `fw.msg_reasoning_only_response.md` user notice, and count them through the stop-unusable-response-loop extension.
1213
- `_30_repeat_response.py` compares `LLMResult.repeat_response_text()` for native calls (public text plus all calls), otherwise full response text. It retries content that exactly matches `loop_data.last_response`, regardless of separate reasoning or provider IDs, using `fw.msg_repeat.md` for history and `fw.msg_repeat_response.md` for the agent-prefixed UI warning text. State advancement is owned by `hist_add_ai_response`; do not call `_remember_llm_result_state` manually.
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
"""Retry model turns cut off by the output token limit with a specific warning."""
2+
3+
from __future__ import annotations
4+
5+
import json
6+
from typing import Any
7+
8+
from helpers import extract_tools
9+
from helpers.extension import Extension
10+
from helpers.print_style import PrintStyle
11+
12+
13+
class TruncatedResponse(Extension):
14+
def execute(self, result_data: dict[str, Any] | None = None, **kwargs: Any) -> None:
15+
if not self.agent or not isinstance(result_data, dict):
16+
return
17+
if result_data.get("skip_default_processing"):
18+
return
19+
20+
llm_result = result_data.get("llm_result")
21+
if not getattr(llm_result, "truncated", False):
22+
return
23+
function_calls = getattr(llm_result, "function_calls", None)
24+
response = getattr(llm_result, "response", "")
25+
if function_calls:
26+
for call in function_calls:
27+
arguments = call.raw.get("arguments")
28+
if isinstance(arguments, str):
29+
try:
30+
arguments = json.loads(arguments)
31+
except ValueError:
32+
break
33+
if not isinstance(arguments, dict):
34+
break
35+
else:
36+
return
37+
elif isinstance(response, str) and extract_tools.extract_tool_request(response):
38+
return
39+
40+
# Name truncation as the cause instead of the generic misformat warning.
41+
warning = self.agent.read_prompt("fw.msg_output_limit.md")
42+
warning_message = self.agent.hist_add_warning(message=warning)
43+
notice = self.agent.read_prompt(
44+
"fw.msg_output_limit_response.md",
45+
finish_reason=getattr(llm_result, "finish_reason", ""),
46+
)
47+
PrintStyle(font_color="orange", padding=True).print(notice)
48+
self.agent.context.log.log(
49+
type="warning",
50+
content=f"{self.agent.agent_name}: {notice}",
51+
id=warning_message.id,
52+
)
53+
result_data["skip_default_processing"] = True

‎helpers/litellm_transport.py‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,7 @@ class LiteLLMTransport:
182182
stop: Optional[list[str]] = None
183183
policy: TransportPolicy = field(init=False)
184184
last_result: LLMResult | None = field(init=False, default=None)
185+
last_finish_reason: str = field(init=False, default="")
185186
last_request_state: str = field(init=False, default=RESPONSES_STATE_PROVIDER)
186187
explicit_prompt_caching: bool = field(init=False, default=False)
187188
history_prefix_hash: str = field(init=False, default="")
@@ -211,6 +212,7 @@ def complete(self) -> ChatChunk:
211212
if self.policy.mode is TransportMode.CHAT_COMPLETIONS:
212213
raw_response = completion(**self._chat_request(stream=False))
213214
parsed = ChatCompletionsTransport.parse(raw_response)
215+
self.last_finish_reason = _finish_reason(raw_response)
214216
self.last_result = self._llm_result_from_chat(
215217
parsed, raw_response
216218
)
@@ -235,6 +237,7 @@ async def acomplete(self) -> ChatChunk:
235237
**self._chat_request(stream=False)
236238
)
237239
parsed = ChatCompletionsTransport.parse(raw_response)
240+
self.last_finish_reason = _finish_reason(raw_response)
238241
self.last_result = self._llm_result_from_chat(
239242
parsed, raw_response
240243
)
@@ -262,14 +265,18 @@ def stream(self) -> Iterator[ChatChunk]:
262265
parser = ChatCompletionsStreamParser()
263266
for chunk in iterator:
264267
parsed = parser.parse(chunk)
268+
if parser.finish_reason:
269+
self.last_finish_reason = parser.finish_reason
270+
self.last_result = self._stream_result_from_chat_parser(parser)
265271
if _has_chunk_delta(parsed):
266272
got_any_chunk = True
267273
yield parsed
268274
parsed = parser.flush()
275+
self.last_finish_reason = parser.finish_reason
276+
self.last_result = self._stream_result_from_chat_parser(parser)
269277
if _has_chunk_delta(parsed):
270278
got_any_chunk = True
271279
yield parsed
272-
self.last_result = self._stream_result_from_chat_parser(parser)
273280
else:
274281
request = self._responses_request(stream=True)
275282
iterator = responses(**request)
@@ -303,14 +310,18 @@ async def astream(self) -> AsyncIterator[ChatChunk]:
303310
parser = ChatCompletionsStreamParser()
304311
async for chunk in iterator: # type: ignore[union-attr]
305312
parsed = parser.parse(chunk)
313+
if parser.finish_reason:
314+
self.last_finish_reason = parser.finish_reason
315+
self.last_result = self._stream_result_from_chat_parser(parser)
306316
if _has_chunk_delta(parsed):
307317
got_any_chunk = True
308318
yield parsed
309319
parsed = parser.flush()
320+
self.last_finish_reason = parser.finish_reason
321+
self.last_result = self._stream_result_from_chat_parser(parser)
310322
if _has_chunk_delta(parsed):
311323
got_any_chunk = True
312324
yield parsed
313-
self.last_result = self._stream_result_from_chat_parser(parser)
314325
else:
315326
request = self._responses_request(stream=True)
316327
iterator = await aresponses(**request)
@@ -452,6 +463,7 @@ def _llm_result_from_chat(
452463
output_items=parsed.get("_output_items"),
453464
provider_model_key=self.model,
454465
capability=self._capability_metadata(),
466+
finish_reason=self.last_finish_reason,
455467
)
456468

457469
def _llm_result_from_response(
@@ -489,6 +501,7 @@ def _stream_result_from_chat_parser(
489501
output_items=output_items,
490502
provider_model_key=self.model,
491503
capability=self._capability_metadata(),
504+
finish_reason=self.last_finish_reason,
492505
)
493506

494507
def _capability_metadata(self) -> dict[str, Any]:
@@ -680,6 +693,7 @@ def __init__(self) -> None:
680693
self.order: list[str] = []
681694
self.emitted = False
682695
self.usage: dict[str, Any] = {}
696+
self.finish_reason = ""
683697

684698
def parse(self, chunk: Any) -> ChatChunk:
685699
if usage := _reported_usage(chunk):
@@ -690,7 +704,9 @@ def parse(self, chunk: Any) -> ChatChunk:
690704
self._append_tool_calls(_get_value(delta, "tool_calls"))
691705
self._append_legacy_function_call(_get_value(delta, "function_call"))
692706

693-
if _get_value(choice, "finish_reason") in {"tool_calls", "function_call"}:
707+
if finish_reason := _finish_reason(chunk):
708+
self.finish_reason = finish_reason
709+
if finish_reason in {"tool_calls", "function_call"}:
694710
text = self._emit()
695711
if text and not parsed["response_delta"]:
696712
parsed["response_delta"] = text
@@ -2222,6 +2238,10 @@ def _first_choice(chunk: Any) -> Any:
22222238
return choices[0] if choices else {}
22232239

22242240

2241+
def _finish_reason(chunk: Any) -> str:
2242+
return str(_get_value(_first_choice(chunk), "finish_reason") or "")
2243+
2244+
22252245
def _get_value(obj: Any, key: str) -> Any:
22262246
if isinstance(obj, dict):
22272247
return obj.get(key)

‎helpers/litellm_transport.py.dox.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
- Fall back to Chat Completions when LiteLLM's Responses mock streaming path tries to JSON-decode a real SSE stream before any output.
3838
- Preserve Chat Completions tool calls from both non-streaming responses and streaming deltas as canonical `LLMResult` function-call items. Non-streaming calls stay canonical when accompanied by commentary; retain that commentary as a separate output message.
3939
- Chat Completions `content_filter` and `refusal` finish reasons raise LiteLLM `ContentPolicyViolationError` on streaming and non-streaming paths; never treat blocked output as an empty success or dispatch partial tool calls. Verify with `tests/test_model_refusal.py`.
40+
- Record the Chat Completions `finish_reason` from streaming and non-streaming responses as `last_finish_reason` and on the resulting `LLMResult`, so callers can tell output-limit truncation apart from misformatted output. Streaming publishes observed finish metadata and canonical calls before terminal or flushed output reaches callbacks, preserving them when a consumer stops early. Verify with `tests/test_model_refusal.py` and `tests/test_litellm_sdk.py`.
4041
- Responses messages explicitly marked `phase: commentary` produce an incremental `response_preview` with public text in `thoughts`, followed by the native tool preview. Previews reuse parsed call state; multiple calls form one parallel envelope. Only native agent-turn callbacks consume the preview: legacy callbacks, response/reasoning deltas, completed output items and call IDs remain unchanged. Unmarked/final-answer text and reasoning are not reclassified as commentary.
4142
- Preserve provider usage and LiteLLM response cost for both transports only when the response or stream actually supplies them; do not synthesize unavailable provider accounting.
4243
- Streaming Chat Completions requests include `stream_options.include_usage` so terminal usage events reach `LLMResult.usage` for OpenAI-compatible endpoints and Messages API providers alike, unless the model configuration previously rejected the option or the request already carries it. A pre-output rejection retries once without the transport-injected `include_usage` and remembers the rejection per model configuration in a process-global cache that naturally clears when the framework process restarts; tests reset it through `clear_transport_capability_cache`. User-provided `stream_options` keys are always passed through untouched, even when the provider rejects them.

‎helpers/llm_result.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
LOCAL_FUNCTION_TOOL_TYPES = {"function_call"}
1010
TEXT_OUTPUT_TYPES = {"message"}
1111
REASONING_OUTPUT_TYPES = {"reasoning"}
12+
TRUNCATED_FINISH_REASONS = {"length", "max_tokens"}
1213

1314

1415
@dataclass
@@ -64,6 +65,7 @@ class LLMResult:
6465
raw: dict[str, Any] = field(default_factory=dict)
6566
capability: dict[str, Any] = field(default_factory=dict)
6667
history_extras: str = ""
68+
finish_reason: str = ""
6769

6870
@classmethod
6971
def from_dict(cls, data: dict[str, Any] | None) -> "LLMResult":
@@ -86,6 +88,7 @@ def from_dict(cls, data: dict[str, Any] | None) -> "LLMResult":
8688
raw=object_to_dict(data.get("raw") or {}),
8789
capability=object_to_dict(data.get("capability") or {}),
8890
history_extras=extras if isinstance(extras, str) else "",
91+
finish_reason=str(data.get("finish_reason") or ""),
8992
)
9093

9194
@classmethod
@@ -133,6 +136,7 @@ def from_chat(
133136
output_items: list[dict[str, Any]] | None = None,
134137
provider_model_key: str = "",
135138
capability: dict[str, Any] | None = None,
139+
finish_reason: str = "",
136140
) -> "LLMResult":
137141
items = [ResponseItem.from_any(item) for item in output_items or []]
138142
if response and not items:
@@ -167,6 +171,7 @@ def from_chat(
167171
state="off",
168172
usage=object_to_dict(usage or {}),
169173
capability=dict(capability or {}),
174+
finish_reason=finish_reason,
170175
)
171176
if not result.response and result.function_calls:
172177
result.response = result.function_calls_text()
@@ -177,6 +182,11 @@ def non_llm(cls) -> "LLMResult":
177182
"""Sentinel for non-LLM AI turns; no response_id, mode/state off."""
178183
return cls(mode="", state="off")
179184

185+
@property
186+
def truncated(self) -> bool:
187+
"""True when the provider stopped because the output token limit was reached."""
188+
return self.finish_reason in TRUNCATED_FINISH_REASONS
189+
180190
@property
181191
def function_calls(self) -> list[ResponseFunctionCall]:
182192
calls: list[ResponseFunctionCall] = []
@@ -233,6 +243,7 @@ def to_dict(self) -> dict[str, Any]:
233243
"raw": self.raw,
234244
"capability": self.capability,
235245
"history_extras": self.history_extras,
246+
"finish_reason": self.finish_reason,
236247
}
237248

238249
def metadata(self) -> dict[str, Any]:

‎helpers/llm_result.py.dox.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
- `from_dict(...)` preserves explicit modes, including the empty non-LLM sentinel; only missing or null modes use the legacy `responses` default.
2323
- `from_chat(...)` must produce an equivalent chat-completions result with `mode="chat_completions"` and `state="off"`, preserving optional function-call output items and provider usage when the chat transport supplies them.
2424
- Function-call output items must preserve `call_id` and optional acknowledged safety checks.
25+
- `finish_reason` carries the provider stop reason for the turn and is not persisted in history metadata; `truncated` is true for output-token-limit reasons (`length`, `max_tokens`).
2526
- `repeat_response_text()` combines response text and canonical calls for native repeat detection, excluding provider IDs, usage and separate reasoning. Non-native replies retain their full response text. This comparison value is transient and does not replace canonical call history or provider metadata.
2627
- Argument parsing must tolerate JSON strings, dictionaries, and malformed values without throwing.
2728

‎models.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -776,6 +776,7 @@ async def unified_turn(
776776
input_items=ResponsesTransport.input_from_messages(msgs_conv),
777777
provider_model_key=self.model_name,
778778
capability=transport._capability_metadata(),
779+
finish_reason=transport.last_finish_reason,
779780
)
780781
if result.output()["response_delta"] and not llm_result.function_calls:
781782
llm_result.response = result.output()["response_delta"]

‎plugins/_context_doctor/AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
- Leave canonical `LLMResult.function_calls` and their accompanying text untouched in either transport. The agent dispatches those calls through the normal tool-policy gate; text repair must not suppress them or reinterpret a textual follow-up as another call.
1919
- Repaired and fallback JSON is always minified.
20+
- Leave results already handled by an earlier `message_loop_result` extension (`skip_default_processing`) untouched, such as output-limit truncation retries.
2021
- Usable Responses output text stays untouched for the core response-tool dispatcher. Recognizable tool intent still enters repair; Chat Completions non-tool output becomes `{"thoughts":[raw]}` and XML-like output becomes `{}` when suppression is enabled.
2122
- Blank-line-separated thoughts expand into separate entries after repair when the split strategy is enabled (default on). This rewrites model-authored thoughts for every repaired turn by design, not only the raw-text fallback branch.
2223
- `_select_value` returns only repaired tool calls or model-authored partial responses; `transform_response` wraps otherwise-raw output as thoughts afterward. `looks_like_tool_call` rejects that synthesized wrapper, so it reaches the fallback branch. Repaired output is usable only with a non-empty thoughts list, headline/tool name string, or tool-args dict.

‎plugins/_context_doctor/extensions/python/message_loop_result/_10_context_doctor.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ class ContextDoctor(Extension):
2121
def execute(self, result_data: dict[str, Any] | None = None, **kwargs: Any) -> None:
2222
if not self.agent or not isinstance(result_data, dict):
2323
return
24+
if result_data.get("skip_default_processing"):
25+
return
2426

2527
# Extract LLM response
2628
llm_result = result_data.get("llm_result")

0 commit comments

Comments
 (0)