docs: fix duplicate word in README configuration section#309
Open
meichuanyi wants to merge 455 commits into
Open
docs: fix duplicate word in README configuration section#309meichuanyi wants to merge 455 commits into
meichuanyi wants to merge 455 commits into
Conversation
…out-distinguishable-from-esc fix(bash): distinguish timeout from ESC-abort in tool_result
…-abort-isolation test(esc): pin async-subagent abort-controller isolation invariants
Brings the Read tool's image handling to full parity with the TypeScript
reference at typescript/src/utils/imageResizer.ts + FileReadTool.ts image
flow. Pillow replaces sharp.
What changes for the user:
- Oversized images (>3.75 MB) are now downscaled to fit the API's 5 MB
base64 limit instead of being rejected outright.
- File extension is no longer trusted blindly — magic-byte detection
sniffs the actual format (PNG/JPEG/GIF/WebP), so a misnamed file.png
containing JPEG bytes gets media_type=image/jpeg correctly.
- Read emits a supplemental isMeta UserMessage carrying image dimensions
("Multiply coordinates by X to map to original image") for
coordinate-mapping prompts.
- The PDF Read with pages="N-M" parameter now extracts page images via
poppler's pdftoppm and routes them through the same image pipeline.
- Bash commands that print data:image/...;base64,... (matplotlib,
mermaid, etc.) surface as image content blocks in the tool_result
instead of garbage text.
- Images >5 MB base64 are rejected pre-API with an actionable error
rather than letting the API round-trip fail.
What changes architecturally:
- New module src/utils/image_processor.py: magic-byte detection,
bounded readFileBytes, maybe_resize_image (3.75 MB / 1568px envelope,
JPEG fallback at q={80,60,40,20}), compress_image_to_byte_budget
(progressive scale × quality, 800×800 PNG palette, 400×400 q=20
ultra-fallback), compress_image_to_token_budget,
create_image_metadata_text.
- New module src/utils/image_validation.py: validate_images_for_api
walks messages, rejects any base64 image >5 MB before the API call.
Wired into claude.py:call_model.
- New module src/utils/pdf_extraction.py: extract_pdf_pages shells out
to pdftoppm, 100 MB input cap, empty-file guard, install-hint error
when poppler-utils is missing.
- New module src/tool_system/tools/bash/image_output.py:
is_image_output / parse_data_uri / build_image_tool_result for shell
image output, with 25 MB pre-decode cap to prevent OOM from hostile
shells.
- src/tool_system/tools/read.py image branch rewritten: bounded read
-> magic-byte sniff -> resize -> token-budget compress -> returns
type='image' with dimensions field + supplemental metadata message.
PDF pages= path wired with try/finally tempdir cleanup. Old
MAX_IMAGE_SIZE_BYTES rejection removed.
- src/query/query.py _dispatch_single_tool now returns
tuple[UserMessage, list[UserMessage]] (primary, extras) so
result.new_messages reach the model. Callers concatenate all
primaries first, then all extras, so multi-tool batches don't break
ensure_tool_result_pairing (a regression the critic caught and the
primaries-first ordering fixes).
- New EventType.IMAGE_PROCESSING analytics event with subtype in data.
Dependency: Pillow>=10.0 (pure-Python wheels on all platforms).
System dependency for PDF page extraction (optional, runtime-detected):
pdftoppm from poppler-utils.
Tests: 60+ new tests across tests/test_image_processor.py,
test_image_validation.py, test_bash_image_output.py,
test_pdf_extraction.py, plus 8 added to tests/parity/test_e2e_file_read.py.
The critical regression test test_multi_tool_batch_preserves_tool_result_pairing
drives _run_tools_partitioned -> normalize_messages_for_api end-to-end to
lock in the primaries-first ordering. Wider parity suite: 442 pass,
4 pre-existing unrelated failures.
Critic review loop completed with APPROVE after two rounds.
See my-docs/image-handling-gap-analysis.md and
my-docs/image-handling-refactoring-plan.md for the full analysis.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ndling-parity feat(read): port TS image-handling pipeline to Python (Tier C parity)
…mage tool_result shape
User-reported correctness bug: the Python build hallucinated wildly when
asked about an @-mentioned image ("a Google search results page") while
TS correctly described it ("an aerial view of a research ship"). Root
cause was two independent bugs both contributing to the failure.
Bug A: ``expand_at_mentions`` opened every @-mentioned file with
``open(path, "r", encoding="utf-8", errors="replace")`` regardless of
extension. For a PNG that produced mojibake (utf-8 replacement chars
over binary bytes) which ``format_at_mention_attachments`` wrapped in a
``<system-reminder>Contents of foo.png:`` block and prepended to the
user message. The model latched onto ASCII fragments (XMP "Screenshot"
metadata, type tags) and hallucinated. Fix: detect image extensions
(png/jpg/jpeg/gif/webp, matching the Read tool's IMAGE_EXTENSIONS)
BEFORE the text-mode open. For image files, run the same image
pipeline the Read tool now uses (post PR agentforce314#154): bounded read ->
magic-byte format sniff -> resize-to-envelope ->
``compress_image_to_token_budget`` fallback when still over the 5 MB
base64 API ceiling. Return a ``kind="image"`` attachment carrying
``base64`` + ``media_type``. New helper ``build_image_content_blocks``
materialises ``ImageBlock`` instances and the REPL composes a mixed
``[TextBlock, ImageBlock, ...]`` user message so the API receives a
real multimodal payload matching the TS auto-Read-on-@image behaviour.
Bug B: ``_dispatch_single_tool`` ran ``json.dumps`` on any non-string
tool_result content, turning Read's properly-shaped image list
``[{"type": "image", "source": {...}}]`` into a text JSON blob. The
Anthropic API then received an image tool_result whose content was
text JSON; the model literally could not see the image and would
hallucinate. PR agentforce314#154's regression test only checked for the synthetic-
error placeholder so this slipped through. Fix: preserve list shape
end-to-end. ``ToolResultBlock.content`` already accepts ``str |
list[Any]``, ``maybe_persist_large_tool_result`` already short-
circuits image lists via ``_has_image_block``, and
``content_block_to_dict`` already serializes per-element -- the
dispatcher just needed to stop coercing.
Collateral fix: PR agentforce314#154's Read tool image content also failed silently
on OpenAI-compatible providers (GLM, Minimax, DeepSeek, OpenRouter)
because ``_convert_anthropic_messages_to_openai`` passed Anthropic
image blocks through unchanged and OpenAI rejects them. New helper
``_anthropic_image_block_to_openai`` translates the base64-source
shape to OpenAI's ``image_url`` data-URI shape. For tool_result image
content the converter now emits ``role=tool`` (text body, with a
placeholder when the original was image-only) followed by a synthetic
``role=user`` message carrying the image_url blocks, since OpenAI's
``role=tool`` doesn't accept multimodal content. Empty-data guard
returns ``None`` rather than producing ``data:image/png;base64,``.
Signature widening to support the multi-block flow:
- ``Conversation.add_user_message(content: str | list[ContentBlock])``
- ``QueryEngine.submit_message(prompt: str | list[ContentBlock])``
Both bodies already supported list content via ``_normalize_message_
content`` / ``MessageContent``; only the type annotations changed, so
existing string-callers (TUI, headless) are unaffected.
REPL UX: image @-mentions skip the direct-stream short-circuit (it
can only carry plain text) and print ``Read image <path>`` instead of
``Listed file <path>`` so the user sees the image was attached.
Test coverage (29 new tests):
- tests/test_at_mention_images.py (17): single + multi-image + mixed
text+image @-mentions, magic-byte detection beats extension, empty/
undecodable images dropped silently, oversize-image compression
brings base64 under API limit (real 4000x4000 random-noise PNG),
doubly-oversize image dropped (monkeypatched), end-to-end through
``normalize_messages_for_api``.
- tests/test_openai_compat_image_translation.py (13): user message
text+image, multi-image, JPEG/PNG/missing-media-type defaults,
empty-data guard, tool_result image-only + image+text + text-only-
no-regression.
- tests/parity/test_e2e_file_read.py (+2): Bug B dispatcher list-
preservation; aggregate-budget pressure path (image still survives
when ``tool_result_chars_so_far = MAX - 1``, counter NOT bumped by
image bytes).
Wider suite: 4984 pass, 0 new failures, 9 pre-existing failures (mcp/
zhipuai import issues + workspace-path tests) unchanged.
Critic review loop: APPROVE after three rounds.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…mention-hallucination fix: @image.png mentions hallucinate; image tool_result stringified
Addresses the four pre-existing gaps the PR agentforce314#155 audit surfaced: 1. validate_images_for_api ran only on the Anthropic-direct call_model path; all provider.chat_stream_response paths skipped it. Promoted into BaseProvider._prepare_messages so every provider validates client-side. query._call_model_sync catches ImageSizeError and surfaces a media_size assistant error (matches the existing server-side classification). agent_loop._call_provider_for_turn re-raises ImageSizeError instead of falling back to chat() (which would re-trigger the same validation). Validator also now walks images nested inside tool_result.content — post-PR agentforce314#154 the Read tool returns image blocks there and the old walker missed them. 2. OpenAI tool->user split symmetric correlation. Tool message body now always carries [multimodal content for tool_use_id=X ...] when the synthetic role=user message follows; the user message leads with a [content for tool_use_id=X] text block so the link is visible from both directions. Empty content sentinel ([empty tool result]) guards the "non-empty content required" invariant for content=[] cases. 3. @file.pdf and other binary @-mentions no longer hit the text-mode open() that produced mojibake-in-system-reminder. Known binary extensions (pdf/zip/docx/...) plus a NUL-byte content sniff route to a new "binary" attachment kind with a Read-tool hint (PDF specifically mentions the pages parameter). BOM-aware decoder (_read_text_with_encoding) handles Windows-emitted UTF-16/UTF-32 files via the BOM-aware codecs (utf-16, utf-32, utf-8-sig) so the BOM is consumed rather than preserved as a literal U+FEFF char. Post-decode garbled-text detector (NUL zero-tolerance + 3% U+FFFD threshold) falls back to a binary attachment when an adversarial spoofed-BOM blob would otherwise leak. 4. _anthropic_image_block_to_openai now has a sibling _anthropic_document_block_to_openai that translates Anthropic DocumentBlock (PDF) to OpenAI's {type:"file"} shape. Defensive: no production path currently produces DocumentBlock for OpenAI-compat, but future ones won't silently pass through. Test plan: - 94 new tests across test_at_mention_binary_files.py, test_base_provider_image_validation.py, test_agent_loop_image_size_propagation.py, test_openai_compat_document_translation.py, plus extensions to test_image_validation.py, test_openai_compat_image_translation.py, test_query_loop.py. - Wider suite: 5015 pass / 10 pre-existing failures unchanged. - Critic review loop: APPROVE after three rounds (round-2 caught a UTF-16 BOM regression I introduced; round-3 fixed it and added the post-decode garble detector). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…dling-audit-followups fix(image-handling): close 4 audit follow-ups from PR agentforce314#155
- Codebase stats: 894 files / 177,428 lines (up from 167,034 on 2026-05-14; ~+10.4k lines in two days). - New news entries: * 2026-05-16 Image-handling parity (agentforce314#149/agentforce314#154/agentforce314#155/agentforce314#156) — Read tool TS image pipeline; @image.png mention fix; image tool_result list shape preserved; OpenAI-compat image/document block translation; validate_images_for_api promoted into BaseProvider._prepare_messages; BOM-aware text decoding. * 2026-05-16 Subagent + Bash reliability — custom subagent discovery from .claude/agents/ (agentforce314#151); Bash timeout vs ESC-abort distinction (agentforce314#152); async-subagent AbortController isolation tests (agentforce314#153); REJECT_MESSAGE on cancelled production path (agentforce314#150). * 2026-05-15 to 2026-05-16 ESC cancellation hardening across providers (agentforce314#144–agentforce314#148) — mid-stream cancel under ~50ms for every provider; shared StreamAbortGuard; LiteLLM worker-thread iteration fix. - Core Systems table: * Multi-Provider description extends to call out Anthropic→OpenAI image/document block translation. * New "Cancellation / Abort" row (✅) capturing the recent sweep. * New "Image Handling" row (✅) capturing Tier C parity. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…pdate-2026-05-16 docs(readme): refresh stats + news + Core Systems for image/ESC work
Bring it in line with the length of the surrounding entries: four facts joined by semicolons, no inline rationale or low-level implementation details. The longer write-up lives in the PR descriptions for agentforce314#149/agentforce314#154/agentforce314#155/agentforce314#156. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…horten-image-parity-news docs(readme): shorten 2026-05-16 image-handling-parity news item
When OpenRouter routes a request to a text-only model (e.g. deepseek/deepseek-v4-pro) and the user @-mentions an image, the provider returns 404 "No endpoints found that support image input". Until now the offending UserMessage stayed in engine and conversation history, so every subsequent text-only prompt re-sent the image and re-triggered the same 404 — the session became unusable. Classify the error in _call_model_sync (mirrors TS's PDF-invalid / image-too-large handlers in errors.ts), tag the AssistantMessage with _api_error="image_unsupported", and surface a friendly message that explains the image was dropped. The engine then strips image blocks from _mutable_messages via the existing strip_images_from_typed_messages helper; the REPL mirrors the strip into session.conversation so the direct-stream path and persisted JSONL stay consistent. TS expects the user to manually rewind via the Ink MessageSelector, which the Rich REPL doesn't have — auto-strip fills that UI gap. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…r-image-unsupported-context-stuck fix(openrouter): break image_unsupported context-stuck loop
The parent agent's live stderr feed during a sync subagent run rendered ``⎿ [type] Name(...)`` for every nested tool use — the literal ``(...)`` was hard-coded, so the user couldn't tell which files the subagent was reading. TS parity surface is ``FileReadTool.ts:369``'s ``getActivityDescription``; the closest Python equivalent (``summarize_tool_use``) already existed but wasn't wired in. Route nested tool_use blocks through ``summarize_tool_use`` via a new ``_format_subagent_tool_use`` helper that flattens newlines, caps at 200 chars, and falls back to bare ``Name`` (not ``Name()``) on empty summaries. 12 tests pin the formatter contract, the call-site wiring, and the legacy ``(...)`` regression. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…gent-read-display fix(agent): show real tool input in subagent progress lines
Gemini's OpenAI-compatible endpoint rejects clawcodex's typical request
shape (content-block arrays without explicit ``type``, JSON Schema with
``oneOf`` / ``additionalProperties``). Native Gemini avoids the compat
layer entirely.
* New ``GeminiProvider`` (``src/providers/gemini_provider.py``):
- Uses ``google-genai`` SDK against
``https://generativelanguage.googleapis.com/v1beta/openai/``
- Converts Anthropic-style messages (text / tool_use / tool_result
content blocks) into Gemini ``Content`` / ``Part`` structures with
role normalization (``assistant`` -> ``model``).
- Recursively sanitizes JSON-Schema tool definitions for Gemini's
strict ``Schema``: drops ``oneOf`` / ``anyOf`` / ``allOf`` / ``not``
(polymorphic params collapse to permissive ``string``),
``additionalProperties``, ``patternProperties``, ``$ref`` / ``$defs``,
conditional / dependency keywords, array extensions, and a few
documentation hints Gemini rejects. Per-tool conversion failures
drop just that tool rather than failing the whole request.
- Auth follows ``src/auth/gemini.py`` (``GEMINI_API_KEY`` /
``GOOGLE_API_KEY`` env or the ``gemini`` provider entry in
``~/.clawcodex/config.json``).
- ``chat_stream`` / ``chat_stream_response`` fall back to non-streaming
for the first cut; ``chat()`` covers the production path used by
headless ``-p`` mode.
* ``PROVIDER_INFO`` and ``get_provider_class`` registered the new
provider name; ``--provider gemini`` is now a first-class CLI value.
Smoke: ``clawcodex -p \"reply hi\" --provider gemini --model
gemini-2.5-pro`` round-trips in <2s. Full SWE-bench Verified run on
Gemini 2.5 Pro resolved 291/499 instances.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drives a side-by-side eval of clawcodex against the openclaude reference
on SWE-bench Verified, using the same model on both sides so the
comparison isolates agent-harness quality.
Workflow (all in ``eval/``):
* ``run_compare.py`` — single-command driver. Spawns each agent as a
FastAPI wrapper (clawcodex_api_server / openclaude_api_server in the
SWE-bench-dev fork), runs ``swebench.harness.run_evaluation`` against
the resulting predictions, diffs the two summary JSONs, writes
``comparison.md`` plus per-bucket triage lists.
- ``--cumulative`` — predictions append, harness scans whole preds
file. Each invocation grows the result; ideal for "start with 50,
add 50 more" iteration loops on the way to full Verified.
- ``--capture-traces`` — wrappers emit stream-json per instance to
``traces/<agent>/<instance_id>.jsonl`` for turn-by-turn analysis.
- ``--predict-workers N`` — concurrent in-flight predictions via
ThreadPoolExecutor with a file write-lock. 3-5x speedup at N=4 if
the model API tolerates it.
* ``pick_batch.py`` — picks the next N unseen instances from a target
dataset given an existing predictions file. ``--random`` for uniform
sampling (recommended for unbiased cumulative estimates) with
optional ``--seed`` for reproducibility; default ``stratified``
rotates one-per-repo.
* ``repair_preds.py`` — offline retrofit for an existing predictions
JSONL: re-extracts the diff, recomputes ``@@ -a,b +c,d @@`` hunk
line counts from the body, ensures trailing newline, rewrites
``model_patch`` in place. Same recomputer is wired into
``run_custom_api.py`` so new predictions are clean automatically.
* ``compare_results.py`` — standalone summary differ used by both the
driver and ad-hoc analysis. Renders the headline table and the
``only_<agent>.txt`` / ``both_solved.txt`` / ``neither_solved.txt``
bucket files.
* ``make_results_chart.py`` — renders the headline result as a
publication-ready PNG (left panel: stacked outcomes per agent;
right panel: per-instance disagreement). Also writes a copy to
``assets/`` for README embedding.
* Operational helpers (``_clear_infra_errors.py``,
``_describe_dataset.py``) for the common "kill stale Docker
image-404 reports and re-run" and "summarize the 500-instance
test set" recipes.
Headline on Gemini 2.5 Pro / full 499-instance Verified split:
clawcodex: 291 resolved (58.2%)
openclaude: 265 resolved (53.0%)
``.gitignore``: ignore ``eval/runs/`` (per-run output GBs of traces +
predictions + harness logs) and stray ``*.patch`` / ``*.diff``
artifacts; gitignore the sibling ``SWE-bench-dev/`` and ``openclaude/``
repos (they live next to clawcodex with independent git histories).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Embed the SWE-bench Verified result chart under the "Why ClawCodex?" section as evidence backing the value props, and add two News entries: the headline benchmark number (291/499 vs openclaude's 265/499 on Gemini 2.5 Pro) and the native Gemini provider landing. The chart (``assets/swebench-verified-gemini.png``) is regenerated by ``eval/make_results_chart.py`` so contributors can refresh it after a new run; the script writes to both the per-run dir and ``assets/``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /tools slash command crashed with AttributeError because ToolRegistry exposes list_tools(), not list_specs(). The /context handler had the same latent bug. Adds a regression test for /tools. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…s-slash fix(repl): use list_tools() in /tools and /context handlers
Tool.description is a Callable[[dict], str] (see build_tool.py:49), but the
/context handler was stuffing the callable directly into tool_schemas,
giving the context analyzer a function reference instead of a string.
Also drop the dead `hasattr(spec, "to_dict")` branch — the Tool dataclass
has no such method.
Reuse the canonical tool_to_api_schema helper so /context and the real API
call site share one source of truth for the {name, description, input_schema}
shape.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ext-schema fix(repl): resolve Tool.description to string in /context schema payload
Foundation layer for the /advisor port from typescript/src. Three
independent pieces, all read-only at runtime until the rest of the
stack lands:
- `src/utils/advisor.py` (NEW): constants (ADVISOR_BETA_HEADER,
ADVISOR_TOOL_TYPE, ADVISOR_TOOL_INSTRUCTIONS), provider/env gates,
block predicates, schema builder, strip_advisor_blocks, and
extract_advisor_{result_text,error_code}. ADVISOR_TOOL_INSTRUCTIONS
is a byte-for-byte copy of typescript/src/utils/advisor.ts:130-145;
a SHA pin plus a live cross-check against the TS source catch drift.
- `src/settings/types.py`: adds `advisor_model: str = ""` to
SettingsSchema. Empty string = unset; future layers read it via
`get_settings().advisor_model` at request time.
- `src/types/messages.py`: extends `ensure_tool_result_pairing` to
strip orphan `server_tool_use` and `mcp_tool_use` blocks whose
matching `*_tool_result` never landed in the same assistant message
(interrupted stream). Mirrors TS messages.ts:5217-5255. Prevents
the API from 400-ing the next turn after a mid-advisor ESC.
## Test plan
- [x] `tests/test_advisor_helpers.py` (32 tests): constants, all gates,
block detection (dict + attribute-style), strip pass shapes
including the empty/thinking-only placeholder fallback, schema
shape, SHA + live TS byte-equality.
- [x] `tests/test_advisor_orphan_pairing.py` (6 tests): advisor orphan
stripped; pair retained; mcp orphan stripped; empty-content
placeholder; regular client tool_use unaffected; multi-message
independence.
- [x] No regressions in adjacent suites (settings, messages utility,
command system, tool normalization, transcript, tui).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…undation feat(advisor) 1/4: helpers + settings field + orphan tool_use guard
…rom TS Closes the parity gap for typescript/src/assistant/ (88 LOC + 11-line stub). Adds: - src/assistant/session_history.py — paginated session-events client mirroring TS sessionHistory.ts (HistoryPage, HistoryAuthCtx, create_history_auth_ctx, fetch_latest_events, fetch_older_events). Preserves the TS four-case body-coercion contract and per-call-site debug labels; bool query params serialize lowercase to match axios. - src/assistant/session_chooser.py — PascalCase stub mirror of AssistantSessionChooser.tsx so a future dialogLaunchers port can import the real name without an alias. - src/assistant/__init__.py — re-exports new names; existing placeholder fields (ARCHIVE_NAME, MODULE_COUNT, SAMPLE_FILES, PORTING_NOTE) preserved. Tests: 30 cases at 100% line coverage. Wire shape (URL path, all five pinned headers, anchor_to_latest=true vs before_id, limit default, 15 s timeout), error policy (4xx/5xx/network/timeout/non-JSON/non-dict all return None; malformed data preserves first_id + has_more), and has_more JS-mirror default are pinned. Signature deviation from TS: create_history_auth_ctx takes access_token + org_uuid explicitly (no prepare_api_request global yet in Python). Matches bridge/code_session_api.py + remote convention. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ing-folder-assistant feat(assistant): port sessionHistory + AssistantSessionChooser stub from TS
Stacked on #PR1. The Anthropic provider's `_build_chat_response`
projects the SDK stream into `(content, tool_uses, ...)` for the
agent loop, but drops everything else — including server-side advisor
blocks. Once the next stack layer wires activation, those blocks need
to round-trip through history so the next turn's API call sees a
matched `server_tool_use(name=advisor)` + `advisor_tool_result` pair.
- `src/providers/base.py`: adds
`ChatResponse.raw_content_blocks: Optional[list[dict]] = None`,
the channel for passthrough blocks the projector chose not to
flatten.
- `src/providers/anthropic_provider.py`: extends `_build_chat_response`
to collect blocks where `type=='advisor_tool_result'` or
`(type=='server_tool_use' and name=='advisor')`, serialize via
`block.model_dump(exclude_none=True)`, and attach to
`raw_content_blocks`. The SDK's lenient `construct_type` preserves
the original fields on unknown discriminators (verified
empirically against `anthropic==0.88.0`), so the round-trip is
faithful.
Other server tools (web_search, code_execution, etc.) are
deliberately NOT scooped up — they have their own SDK projection.
## Test plan
- [x] `tests/test_advisor_chat_response_roundtrip.py` (6 tests):
builds a real SDK ParsedMessage via `accumulate_event` for
each of the three discriminated `advisor_tool_result` content
shapes (`advisor_result`, `advisor_redacted_result`,
`advisor_tool_result_error`), the orphan case, the non-advisor
server-tool exclusion, and a mixed-block case (text + tool_use
+ advisor pair).
- [x] Pins the SDK round-trip contract — a future SDK upgrade (e.g.
Pydantic v3, stricter discriminator handler) that silently
drops extra fields on unknown discriminators would fail the
`encrypted_content` and `error_code` assertions locally.
- [x] No regressions in adjacent provider tests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stacked on #PR2. With helpers + provider preservation in place, this
is the actual server-side parity wiring — same beta header, same
schema, same instructions, same cache-preserving order as TS
typescript/src/services/api/claude.ts.
- `src/state/app_state.py`: adds `advisor_model: str | None = None`
field + `_on_advisor_model_change` handler. The handler persists
to `settings.advisor_model` via the shared default ConfigManager
AND invalidates the settings cache, so mid-session toggles via the
reactive store reach the next API call without a process restart.
Registered in `_FIELD_HANDLERS` to satisfy the coverage contract.
- `src/query/query.py`: `_call_model_sync` now:
1. Computes `advisor_active` via a single try/except wrapper:
`is_advisor_enabled(provider)` (first-party Anthropic AND env
not disabled) AND `get_settings().advisor_model` non-empty AND
`model_supports_advisor(provider.model)` AND
`is_valid_advisor_model(candidate)`. Any exception → inactive.
2. When inactive, strips historical advisor blocks via
`strip_advisor_blocks` (the API 400s on advisor blocks without
the beta header).
3. When active, appends the advisor schema AFTER the cached tool
list (preserves the `cache_control` marker position), sets
`betas=[ADVISOR_BETA_HEADER]`, and appends
`ADVISOR_TOOL_INSTRUCTIONS` to the system prompt's tail. The
append order mirrors TS claude.ts:1395, 1411-1421 — toggling
/advisor doesn't bust the cache prefix.
4. After the response, preserves `raw_content_blocks` (from PR2)
into `assistant_blocks` so the pair survives history replay.
A stale `advisor_model` setting under a non-supporting base model is
silently ignored — never sent to the API. Mirrors the TS "skipping
advisor — base model X does not support advisor" branch.
## Test plan
- [x] `tests/test_advisor_request_wiring.py` (12 tests): full
activation matrix on a first-party Anthropic provider, plus
defensive activation (synthetic exception in the gate → turn
proceeds without advisor). Verifies the beta header attaches,
schema appends AFTER regular tools, instructions append to
both string and block-list system prompts, advisor blocks
survive in history when active. Negative cases: env disabled,
custom base_url, unsupported base model, invalid advisor
model, settings unset — none send the schema/header.
- [x] `tests/integration/test_advisor_smoke.py` (3 tests): mocked
Anthropic stream produces text + server_tool_use(advisor) +
advisor_tool_result + text — verify pair preserved across
turns. Interrupt scenario: orphan use stripped on next-turn
replay even when beta IS active (orphan strip is unconditional
because the API rejects orphans regardless of header).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…lows-two-pane-tui Interactive two-pane /workflows TUI (Phases ⟷ Agents)
Replace the binary Allow/Deny permission surface with the TS PermissionPrompt option set — allow once / allow always (persisted rule) / deny / deny with feedback — and un-sever the engine: - PermissionAskRequest/Reply protocol replaces the legacy 3-arg handler on all four producers (TUI bridge, REPL console menu, headless auto-deny, subagent share); real tool_input now reaches the modal so per-tool previews render - registry applies chosen updates to the live context and persists via the new settings_paths resolver; setup_permissions is wired into all three entrypoints so persisted rules load at startup - bash "don't ask again" suggestions ported from TS heuristics (2-word prefix, heredoc, safe env vars; no LLM extractor) - engine fixes: multi-word prefix rules now match (TS bashPermissions.ts:879-882 semantics); new quote-aware chaining guard stops prefix/wildcard rules from auto-allowing compound commands (documented divergence — Python matches whole strings) Suite at the verified main-0fa986f baseline (12 pre-existing failures); +31 new tests incl. the persist→reload→auto-allow restart round-trip. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Typing y/n/a/d into the focused feedback Input must reach the Input, not the screen bindings — pilot-verified, now regression-locked. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- no-wildcard rules now match exact-or-word-boundary, closing the last-token elongation hole the new exact suggestions activated - multiline/compound derivation can no longer mint dead rules (chained first line -> 2-word prefix or nothing) or bare-shell prefixes; heredoc at index 0 yields nothing - chaining scanner: lone & detected as a separator (redirections >&, <&, &> skipped); ANSI-C quoting documented as a non-goal - env-assign regex pinned to ASCII (JS \w parity); import hoisted out of the matcher hot path; __all__ completed; +13 tests locking the above incl. the accepted basename-normalization trade-off Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Read "don't ask again" suggestions removed: the engine consults
content rules only for Bash, so the persisted Read(<dir>/**) rule
could never match — re-asking right after promising not to is
worse than no option (path-rule matcher = parked follow-up)
- project/local permission settings move to .clawcodex/ — the
.claude/settings{,.local}.json names are owned by the real Claude
Code harness; sharing them would read AND mutate the harness's
live rules (this worktree's own .claude/settings.local.json
proved the collision)
- registry apply/persist failures now logged (incl. per-update
persist result); subagent rebind divergence documented at the
rebind site; _with_default_suggestions returns a replace() copy
instead of mutating tool-owned decisions
- protocol surfaces properly typed (TYPE_CHECKING imports); dead
_preview_tool_input removed; stale doc-gate comment corrected;
chosen_updates None-safe; defaultMode write-only asymmetry noted
- tests: REPL menu mapping (always/feedback rows, enable-row
numbering shift, no-cache-on-always, rebind-target identity),
headless PRODUCTION setup block loads persisted rules, autouse
user-settings isolation fixture (hermeticity)
Suite re-verified at the main-0fa986f baseline (12, identical set).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-permission-options components C1: multi-option permission flow wired to the rule engine
The dormant resume screen becomes real: /resume (alias /continue) lists persisted sessions and selection swaps the live conversation. - listing/filtering live in UI-neutral services/session_listing (headless /resume loads zero Textual modules); metadata-only sessions (message_count==0) are filtered with an honest hidden count per the standing critic condition; duplicate ids deduped - AgentBridge.resume_session: busy-gated swap using the TS-parity reader (orphan repair, snip boundaries, path adjustment), then switch_session + restore_cost_state_for_session in TS lockstep (ResumeConversation.tsx:224-227), advisor scan cursor pinned to the list tail (index 0 would replay historical advisor events), persister re-targeted (title/counters preserved); unreadable transcripts refuse instead of crashing the callback chain - degraded replay renders dict AND dataclass text blocks (the reader yields TextBlock dataclasses) and counts skipped tool/attachment blocks honestly; session-preview pane explicitly parked in the module docstring - registry command: TS-verbatim metadata incl. aliases=[continue], substring search-term filter, output-style pattern (no ctx.ui) 19 resume/doctor tests (incl. a production-pipeline replay test and the persister round-trip asserting title survival + no count double-bump); full suite baseline-identical (12 known failures). Critic-approved after one REQUEST CHANGES round (B1 dataclass-block render bug, dependency direction, advisor replay, cost restore). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-resume-picker components C2: resume picker wired end-to-end
Status/context half of plan §3 (C3b ships the transcript polish):
- context-% status segment + context-low transcript row, driven by
the LAST response's prompt-side tokens (new last-wins last_* keys
in the compat usage incl. cache read/creation — the cumulative
sums double-count and drop cache, TS tokens.ts:407-420); zeroed
on /clear and /resume
- threshold math delegates to the CANONICAL autoCompact port the
engine already uses (review caught a forked module warning at the
engine's compact point — the fork is gone); the canonical percent
display modernized to current-TS raw-window half-up rounding;
warning copy branches on auto-compact enabled ("N% until
auto-compact") vs disabled ("Context low … /compact")
- settings statusLine.command runner (TS executeStatusLineCommand:
JSON payload on stdin w/ TS field shapes, 5s timeout, failures →
no text); event-driven refresh on run-finish + 30s hot-reload
keepalive instead of hot polling; thread worker hardened so no
codec/teardown error can exit the app
20 new tests (thresholds DERIVED from the canonical functions and
pinning warn < compact < window); suite baseline-identical.
Critic-approved after one REQUEST CHANGES round (usage measure,
threshold fork, worker crash vectors, payload shapes, hot polling).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-context-status components C3a: context awareness + custom statusline command
Transcript half of plan §3, completing C3: - compact boundary row: typed plumb (LocalCommandResult "compact" → CommandResult → dispatch compact flag → distinct boundary row); legacy REPL gained the compact elif so the executed command never falls through to re-handling - ctrl+o expand-last-truncated (legacy _expandable_blocks parity): tool results exceeding the panel limits stash full text under the ROW's authoritative name — production result events carry tool_name="" (review B1; all tests now use that shape) - read-group collapse: ≥3 consecutive completed Read/Grep/Glob rows fold into one summary row; breaks on user/assistant/thinking/ advisor/system rows, non-read tools, and errors; ONE ctrl+o stash entry per group tracked by identity (per-read content stashes interleave — the positional guard flooded the deque, review M2); boundary/group/expanded rows participate in the post-exit scrollback dump via _SnapshotStatic (review M3) - /thinking session toggle wired REAL: bridge → compat → QueryParams.extended_thinking; enable refuses honestly unless the provider+model actually support it (explicit True bypasses the query gate — review M4); first use disables (TS ThinkingToggle) 14 tests on production-shaped events; suite baseline-identical (the one transient delta re-verified as the documented prompt-assembly second-boundary flake, 3/3 isolated passes). Critic-approved after two rounds. Known cosmetic: after the first fold the group row stays pinned while later read rows flash below it briefly — closest achievable to TS static-region grouping. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…b-transcript-polish components C3b: transcript polish + /thinking toggle
Port of TS processBashCommand: a user-typed !command runs DIRECTLY
through the Bash tool call (no registry permission flow — user-typed
commands aren't model-initiated; the tool's defense-in-depth
dangerous-pattern guard still applies and is pinned by a test that
also asserts the permission handler is NEVER consulted), produces no
agent turn, and feeds <bash-input>/<bash-stdout|stderr> user messages
into the conversation so the model sees them next turn.
- conversation integrity (review B1): texts route through the new
AgentBridge.append_user_texts — deferred while a run is in flight
(mid-run appends could interleave a tool_use/result pair, durably
via the persister) and drained in _finish() before the durable
flush, plus at the top of submit() so teardown-window stragglers
land BEFORE the next prompt; this also replaces the app's
_persister reach-in
- UI: echo row mounts synchronously ("running…"), the worker fills
the output in place; BOTH streams render (git/npm write to stderr
with exit 0); shared truncate_body() so panel/stash limits can
never drift; ctrl+o-expandable; sequential-only with honest
refusals (busy agent / bash already running); magenta input accent
on the ! prefix
- divergences documented: no caveat message, both streams escaped,
no PowerShell branch, no live progress/ESC cancel yet (follow-up)
15 tests incl. deferral through a real bridge, exit-143 timeout via
env shrink, !cd cwd-persistence pinned as deliberate TS parity, and
repl dispatch (bash called, agent NOT). Suite baseline-identical.
Critic-approved after one REQUEST CHANGES round.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-bash-mode components C4: bash-mode (!) input
/search and /open dialogs (TS GlobalSearchDialog / QuickOpenDialog, degraded: no preview pane, command-launched instead of ctrl+shift chords), inserting TS-verbatim @file#Lline / @path mentions into the composer — and the #L fragment now ATTACHES the file downstream (expand_at_mentions strips it; range slicing is a noted follow-up). - shared-engine bugfix: the ripgrep abort-poll runner never drained its pipes, so >64KB of output deadlocked rg until the 20s timeout (rg --files tripped it first; big content searches had the same latent stall); now drained via concurrent daemon threads, with errors="replace" so one non-UTF8 matched line can't kill the drain and silently truncate Grep/Glob results - UI-neutral services: workspace_search (TS-cited caps 10/500, -i -F, the TS Windows-safe ^(.*?):(\d+):(.*)$ parse, POSIX-relative insertions, truncation flags) and fuzzy_match (fuzzy_score moved verbatim out of the Textual history module, which re-exports it) - screens on the house DialogScreen+SelectList idiom: Input keeps focus, arrows route to the list, Enter selects; stale-result race closed in depth (immediate clear + query guard + per-search abort tripped on supersede/close + is_attached); honest counts incl. truncation labels; aborted workers never report 14 tests incl. the >64KB no-stall regression, the stale-enter pilot, case-insensitivity, truncation flags, and the functional-mention pin. Suite baseline-identical. Critic-approved after one round; known pattern-level wart shared with HistorySearch (mouse-focused list swallows enter/esc) noted for the shared-idiom follow-up. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-search-dialogs components C5: workspace search dialogs + ripgrep drain fix
- /doctor finally reaches the dormant DoctorScreen (TUI inversion) and a new output-style registry command serves headless/REPL (python/ platform, ripgrep availability, sessions dir, config + rule health) - services/config_health: per-file JSON/encoding/top-level checks over the paths the loaders ACTUALLY read (git-root-anchored project config via get_project_config_path — review M1 — plus the .clawcodex settings trio and the managed file); startup warning rows in a new visually-distinct "-warning" SystemMessage variant, with "Run /doctor for details." - loaders HARDENED so "file ignored" is true for every detected class (review M2): permissions/setup._load_settings_file catches ValueError/OSError + rejects non-dict top levels (a mis-encoded or array-shaped settings file crashed setup_permissions before the TUI mounted); config._read_json rejects non-object top levels (which previously blew up in _deep_merge) - delivers the C1-deferred rule-warning surfacing (review M3): collect_rule_warnings formats dangerous + shadowed permission rules into the startup rows, the /doctor report, AND a new DoctorScreen section; the stale C1 comment now states it - conftest: GLOBAL_CONFIG_DIR isolated per-test (full-app tests no longer read the developer's real global config) 18 tests incl. loader-hardening regressions through the REAL loaders, a real-CommandContext pin, unreadable/dir-shaped paths, and a subprocess textual-free import check. Suite baseline-identical. Critic-approved after one round; KeybindingWarnings deferral and the no-reset-gate divergence documented in the module. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-doctor-validation components C6: /doctor wiring + config-validation surfacing
…alog) Repo-declared MCP servers (.mcp.json, scopes project+local) are now untrusted until the user approves them, porting TS getProjectMcpServerStatus (utils.ts:351-390) + MCPServerApprovalDialog: - services/mcp_approval.py: status precedence disabled->rejected, enabled->approved, enableAll->approved, else pending; merged user->local settings read (lists unioned, scalars last-win; project tier deliberately excluded so a repo cannot self-approve - documented safer-than-TS deviation); choices persist to .clawcodex/settings.local.json. - Enforcement at both choke points: get_all_mcp_configs filters .mcp.json scopes PRE-merge (a name-colliding unapproved repo server can no longer shadow-then-drop a user-scope server - critic-reproduced bug), and get_mcp_config_by_name gates per-name resolves (reconnect/OAuth side door). Pending servers warn via the existing ValidationError channel; rejected servers drop silently. - TUI: McpApprovalScreen (enable / enable all / disable; Esc = decide later) chained at boot per pending server; enable_all short-circuits the queue; write failures report honestly and keep servers pending. - Elicitation declared GATED: the MCP client has no server->client request support, so ElicitationDialog parity stays parked with the client work (gap doc T12 corrected). 20 new tests incl. cross-scope shadowing regression and screen pilot coverage; full suite baseline-identical. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-mcp-approvals components C7: .mcp.json server approval gate
…pass) Three TS-verified boot gates, chained strictly sequentially before the C6 config warnings and C7 MCP approvals: - Folder trust (TrustDialog.tsx, config.ts:771-817): per-project hasTrustDialogAccepted in the USER-owned global config's projects[path] map (git root else cwd) - never a committable file, so a repo cannot pre-trust itself; parent trust covers children; home-dir acceptance is session-only; decline exits 1. Warning enumeration degraded to the two subsystems this port models (bash allow rules + statusLine). config.py grows get/update_project_entry + get_project_path_for_config. - External CLAUDE.md includes (ClaudeMdExternalIncludesDialog.tsx, claudemd.ts:805-815/1427-1437): asked once per project via the boolean pair hasClaudeMdExternalIncludesApproved/WarningShown; get_memory_files now derives include_external from the approval and keys its cache on the effective value; is_external_memory_file excludes the User tier exactly like TS; Esc maps to "no". - Bypass acceptance (BypassPermissionsModeDialog.tsx): decline exits 1, Esc exits 0 (distinct TS codes, now propagated - run_tui returns app.return_code); accept persists skipDangerousModePermissionPrompt to user settings and syncs bootstrap session trust state. - C7 deferred item closed: get_mcpjson_server_status ports the TS auto-approve branches (bypass-accepted flag; non-interactive session, utils.ts:377-403) - headless/SDK runs now match TS instead of the interim skip-with-warning; run_tui defensively pins set_is_interactive(True). Critic loop: REQUEST CHANGES (1 MAJOR: forked session-trust state + stale init.py TODO; 5 MINOR) -> all fixed -> APPROVE. 22 new tests; full suite baseline-identical. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-startup-gates components C8: startup security gates (trust / includes / bypass)
A prompt starting with '#' saves a note to a chosen memory file instead of starting an agent turn: repl routing (precedence /, !, #; bare '#' falls through) -> MemorySaveScreen over the /memory target hierarchy (build_memory_options, promoted to a public shared API so the two pickers can never drift) -> services/memory_append.py. Honesty note (documented in the plan, gap doc, and module docstring): the vendored TS snapshot ships only the RENDERER (UserMemoryInputMessage.tsx - '#' row + random "Got it. / Good to know. / Noted.") and the selector; no '#' detection or append exists there. TS-verbatim pieces are ported verbatim (acknowledgement trio, "Cancelled memory editing", target hierarchy); the connective glue is explicitly port-derived: ensure-create + single-newline separation + '- ' bullet unless the note supplies its own punctuation; every failure/cancel path restores '#note' into the prompt so nothing typed is lost; notes are recorded in prompt history like '!' commands. Critic: APPROVE (3 MINOR / 4 NIT, all applied: docstring precision, history recording, enumeration-failure test, ValueError guard, UIOption typing, public build_memory_options). 17 tests; full suite baseline-identical. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-memory-shortcut components C9: '#' memory-append shortcut
… run finishes
Background workflows enqueue a <task-notification> envelope on completion, but
nothing in the REPL ever drained that queue — its own docstring admits the
consumer ("WI-3.3") was deferred. So a finished /deep-research run surfaced only
by polling /workflows, with no completion signal and no pointer to the results.
Wire the consumer:
* src/repl/task_notifications.py (new, pure/tested): deterministic completion
banner from the terminal task state (✔/✗/⊘ · name · agents · tokens · duration
· run journal), an envelope fallback, and build_notification_turn() which wraps
drained envelopes in a guiding system-reminder for the agent.
* REPL glue (core.py): _deliver_pending_task_notifications() drains the queue at
the turn boundary, prints each banner, then feeds the envelopes to the agent as
one turn so it reads the result and summarizes conversationally — the Claude
Code "the research is done…" behavior. A daemon watcher wakes the idle ❯ prompt
the moment a run finishes (verified cross-thread against prompt_toolkit 3.0.x
via app.exit), so it surfaces without a keystroke — and never clobbers a
half-typed line or a permission dialog.
Tests (15): pure helpers, the REPL delivery method (banner + single agent turn,
no-op on empty queue), the wake guard (idle+empty exits; half-typed/not-running
don't), the watcher (wakes when idle+pending, silent otherwise), and an
end-to-end run through the production launcher → enqueue → REPL delivery with a
fake runner. Pre-existing test_workspace_search_c5 failures (ripgrep env) are
unrelated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…low-completion-notifications Notify on background-workflow completion (banner + agent summary)
…oring keyword Closes two gaps in the workflow-engine port (workflow-engine.md §4.1 + §4.7), audited and planned in docs/workflow-commands-and-ultracode-plan.md. 1. Dynamic workflow slash commands were discovered but never dispatchable. load_workflow_commands fed only the aggregator's get_commands(), which has no real REPL consumers, so saved .claude/workflows/*.py never reached the global registry that dispatch + suggestions read (the same orphaning that hid /deep-research until agentforce314#267). Add load_and_register_workflows(registry=None) — mirroring load_and_register_skills, same shadowing guard (builtins/bundled win; project beats personal) — and call it at REPL (_init_command_system) and TUI (app.py) startup. Saved workflows now dispatch AND autocomplete (workflow tag). Confirmed end-to-end: a saved file that was `dispatch: False` is now registered, suggested, and tagged. 2. The `ultracode` authoring keyword (§4.1) was never built (0 hits in src/). New src/workflow/ultracode.py: the standalone `\bultracode\b` keyword in a prompt (one-shot) and `/effort ultracode` (session-long mode) append a <system-reminder> nudging the model to author a workflow via the Workflow tool instead of working turn by turn. Wired into REPL chat() (next to the companion-intro append) and effort_command.py. Gated by is_workflows_enabled() — §4.8: the keyword no-ops and the option leaves the /effort menu when off. (Python's effort pipeline is inert, so /effort ultracode contributes the orchestration mode only, not a reasoning level.) Tests (48 new): saved-workflow registration + precedence + gating (test_workflow_dynamic_commands.py); keyword detection, reminder precedence, session toggle, and /effort ultracode (test_ultracode.py). Three base-effort picker tests gated to workflows-off so they keep asserting the bare option set (ultracode-in-menu is covered separately). Full sweep: 299 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…low-commands-and-ultracode Dispatchable saved /<name> workflows + the ultracode authoring keyword
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
infrom the README configuration section.Testing