Repository navigation
Suspend project: drop xbench overfitting, fix license notices, add status docs - #7
Merged
Merged
Conversation
Add /v1/chat/completions and /v1/models endpoints that translate between the OpenAI Chat Completion format and zen-claw's agent loop, enabling evaluation harnesses like xbench-evals to call zen-claw via the standard openai Python SDK. - New module zen_claw/api/openai_compat.py with FastAPI router - Support both non-streaming (JSON) and streaming (SSE) responses - Mount router in dashboard server alongside existing webhooks - 16 tests covering completions, streaming, models, error handling Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `eval` optional dependency group with openai SDK - Add scripts/xbench_serve.ps1 one-click evaluation launcher - Create release/xbench-v1 branch as evaluation baseline Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix _invoke_agent_text: pass web_search_config, web_fetch_config, and brave_api_key so search tools work via dashboard API - Improve message handling: prepend system messages to user content, enabling eval harnesses to pass format instructions - Add test for system message prepending behavior Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- tests/eval/xbench_regression.py: 17 regression tests covering schema compliance, resilience, xbench prompt format (最终答案), streaming, and 50-call throughput stability - scripts/xbench_patch.py: idempotent patch script that adds zen-claw to xbench-evals/language_models.py (API_ENDPOINTS, API_PRICE, match case) Run before every xbench submission: pytest tests/eval/xbench_regression.py -v Integrate with xbench-evals: cd xbench-evals && python ../zen-claw/scripts/xbench_patch.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. direct_contracts: json_format contract required trigger keyword
- `_extract_json_format` was matching any message containing `[...]`
or `{...}` without a trigger keyword, so xbench prompts like
`[问题]: 地球距太阳约多少天文单位？` were silently handled as JSON
format requests, returning "Invalid JSON: ..." instead of the LLM.
- Fix: require `_JSON_TRIGGER` to match before extracting a JSON body.
2. server: _invoke_agent_text was missing tool_policy_config
- `AgentLoop` defaulted to `ToolPolicyConfig()` whose
`production_hardening=True`, causing web_search / web_fetch to be
denied with "disabled until domains are explicitly allowlisted".
- Fix: pass `cfg.tools.policy` so the config-file setting takes effect.
3. loop: forced synthesis when max tool iterations exhausted
- When a model keeps calling tools without producing a text response
the loop returned None, yielding "I've completed processing but have
no response to give." For free-tier models (MiniMax, Nemotron) this
was the common path.
- Fix: after the while loop, if final_content is None, make one final
LLM call without tools plus strip any XML tool-call artefacts from
the response.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…or handling - web.py: add DuckDuckGo provider via ddgs package with 3-attempt exponential backoff (2/4/8s) on rate-limit errors - schema.py: add "ddg" to WebSearchConfig provider Literal - server.py: strengthen _DEEPSEARCH_SYSTEM_PROMPT with mandatory final-line format requirement; replace silent [echo] exception swallowing with proper logging and [error] response - context.py: revert memory recall logic to preserve long-term memory display when recall_mode="none" (only skip query-ranked lookup) - tests: update expected error code brave_api_key_missing → search_api_key_missing to match generic multi-provider check Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Codex <codex@users.noreply.github.com>
Co-authored-by: Codex <codex@users.noreply.github.com>
Keep: reasoning_content passthrough for thinking-mode providers, hosted_vllm prefix de-duplication, web_fetch HTML fallback, failed-fetch and query/URL trace fields, xbench direct-router bypass, fetch-first follow-up nudge, xbench-only forced synthesis, per-request xbench session isolation, short-answer formatting, and the eval runner/patch/regrade scripts. Drop: hardcoded per-question answers (~95-entry prompt->answer table), question-specific extractors, rerankers and query enrichment, and the tests that asserted them. The full original WIP is preserved on the local branch archive/xbench-tuning-2026-05. Earlier 10/10 and 33/100 xbench scores are marked invalid in tasks/done.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o feat/phase-c-unified-gating
- README: add bilingual maintenance-status section (suspended since 2026-10-23, test baseline, no real-world evidence, withdrawn xbench scores, known tech debt, where to resume) and nanobot attribution. - NOTICE: repository modifications are BSL 1.1, not MIT; reproduce the upstream nanobot MIT copyright and permission notice as MIT requires. - Remove links to docs that are gitignored and therefore missing on GitHub, plus the reference to the deleted docs/backlog.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gating # Conflicts: # README.md # docs/architecture-and-design-principles.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wrap-up before suspending the project: brings the April xbench work, the cleanup of benchmark overfitting, and the suspension docs into
main.Code
scripts/xbench_*).reasoning_contentpassthrough for thinking-mode providers,hosted_vllm/prefix de-duplication,web_fetchHTML fallback, failed-fetch / query / URL trace fields, xbench direct-router bypass, per-request xbench session isolation, short-answer formatting.archive/xbench-tuning-2026-05, which was not pushed.Docs / licensing
docs/backlog.md.mainin: conflicts inREADME.mdanddocs/architecture-and-design-principles.mdwere resolved in favour of this branch.main's versions were identical to this branch's June commits, which this branch only extends.Test plan
ruff check .passes1816 passed, 39 skipped.tests/test_browser_sidecar_integration.pywas excluded: Playwright cannot launch a browser on the local machine.zen_claw/,tests/,scripts/finds no remaining hardcoded benchmark answers🤖 Generated with Claude Code