Skip to content

fix(noema): expose structured failure kind - #1898

Open
seonghobae wants to merge 65 commits into
mainfrom
codex/noema-failure-kind-telemetry
Open

fix(noema): expose structured failure kind#1898
seonghobae wants to merge 65 commits into
mainfrom
codex/noema-failure-kind-telemetry

Conversation

@seonghobae

@seonghobae seonghobae commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Current integration receipt — 2026-09-05

Exact head: a7afb39d8c9e6b1b20857e378a236f566c94ff36; tree: fdf57270e965050f7c781ef40303af48d5e6dd3c; protected base: f250638827f8252b0d9e5cb2601f4d333f96162f. The existing df0f735f42adbb44d45f3c3a4e503e400b47ed79 failure-kind delta and the merged #1922 prerequisite are preserved by ordinary merges. This same PR branch was pushed without force; no closure, approval, ruleset change, or merge was performed.

Causal repair and evidence boundary

The original patch handles optional error.detail.failure_kind. Protected contextual-orchestrator a080297d2546bb61e89520d637cabc202db331ec already returns error.code=invalid_structured_output for a structured-response error but that path has no failure kind, model, or attempt list. The consumer was discarding this existing classification. It now retains canonical error.code independently, using the same bounded reader and identifier validator in both the single failure annotation and the raised diagnostic.

CO #1004's typed exhaustion detail remains proposed, not a released dependency or proof of the cause of Naruon #1244's historical 502. Noema still makes one gateway request and fails on the returned error; no provider routing, paid fallback, retry, timeout, or semantic-approval rule changed. Identifier format checking is not arbitrary secret detection. Free-form messages, request identifiers, and unrelated body fields remain excluded.

The existing doctoring record and G-02/G-03 baseline carry exact producer/API/log links, alternatives, OWASP APA 7 grounding, regression history, ownership, and remaining delivery gates. CHANGELOG records the observable maintenance benefit without claiming incident resolution.

Verification on the unchanged candidate

  • RED: removing only canonical-code extraction/formatting yielded 8 failed / 43 passed. Restored field matrix: 51 passed, including the sparse current gateway envelope, malformed values, 200/201-character boundaries, CR/LF, escapes, surrogates, absent siblings, one request/annotation, and no unrelated payload text.
  • Focused Noema/edge/environment regression: 176 passed.
  • Final normal full suite: 2940 passed, 1 skipped, 21 subtests passed (115.62 s).
  • Final GITHUB_ACTIONS=true full suite: 2940 passed, 1 skipped, 21 subtests passed (141.22 s). Five measured modules: 1512 statements / 606 branches, 100%; Noema alone: 854 statements / 384 branches, 100%.
  • Noema docstrings: 100%. Ruff 0.15.21 and compilation passed; unused test import found by Ruff was removed. Worktree was clean and head/tree unchanged before and after verification.
  • The existing reviewed-LLVM-19 admission case is skipped on this macOS host because the required tools are absent. It is unverified, not passed.
  • Task-local Python 3.12.13 uses the hash-locked review requirements plus the separately declared pip==26.2.1 tool. The initial missing-pip failure was corrected in the environment, not hidden or converted into a source change. This is not claimed as a fully hash-locked clean install or hosted Python 3.14 execution.
  • Independent read-only review identified the missing sparse-envelope case; it was added and re-reviewed with no remaining actionable finding. This is not a qualifying GitHub approval.

Commands (repository root, task-local environment):

PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pytest tests -q -W error -rs
GITHUB_ACTIONS=true PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pytest tests -q -W error -rs --cov=scripts.ci.noema_review_gate --cov=scripts.ci.pr_review_conflict_scope --cov=scripts.ci.pr_review_autofix_context --cov=scripts.ci.zdr_policy --cov=scripts.ci.contextual_orchestrator_review_policy --cov-branch --cov-fail-under=100
.venv/bin/python -m interrogate --fail-under 100 scripts/ci/noema_review_gate.py

JUnit SHA-256: normal 9cec553a5a503cbf1ba529eafdb79f8b82951aa978c2fa232816ede3d44608a5; CI be8f30f2c08c451c747f3a2c3def264674cd47c8fb6fc3eca7f77d7ae136c881. Coverage JSON SHA-256: 3ccad4837f674d8028e4ce83cbe83e1a95ea12eb45f283fb0d11ee3ac7e4df7e.

Remaining protected-delivery work

New exact-head quality run 33962985324, Noema run 33962984599, security scans, and independent review are queued/pending after the push. Preserve those handles and re-fetch terminal evidence; do not repeat dispatches or borrow old-head passes. The separate maintainer exception reported for #1922 does not authorize bypass for this PR. Local checks do not prove protected merge, release, real PostgreSQL/browser operation, or a deployed gateway fix.

Historical original receipt

The text below describes the original df0f735 proposal and its 123-test run. Its mention of CO #1004 is proposed-contract context, not a claim that the current protected structured-error path already emits that field.

문제

contextual-orchestrator가 안전한 HTTP 오류 detail에 failure_kind=structured_output_exhausted를 제공해도 중앙 Noema는 해당 필드를 allowlist하지 않아 Actions 로그에서 구조화 응답 실패 원인을 버렸습니다.

수정

  • 기존 bounded HTTP error parser에서 안전한 식별자 규칙으로 failure_kind를 수용합니다.
  • 공개 로그에는 allowlist된 scalar만 기록하며 원문 응답과 임의 필드는 계속 폐기합니다.
  • HTTP 502 회귀 테스트에 structured_output_exhausted 전달과 비밀 비노출을 검증합니다.

검증

  • uv run pytest -q tests/test_noema_review_gate.py tests/test_noema_model_output_edge_coverage.py
  • 123 passed
  • git diff --check

#1004가 제공하는 typed structured-output detail과 중앙 Noema telemetry를 연결합니다.

@seonghobae
seonghobae enabled auto-merge (squash) September 5, 2026 05:41
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 5, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-05T05:44:15.739553Z df0f735 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 34 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: b95ead3e-3842-4592-80d9-2228ab0ed325

📥 Commits

Reviewing files that changed from the base of the PR and between 6f8c51d and f1cd875.

📒 Files selected for processing (10)
  • CHANGELOG.md
  • docs/doctoring/noema-observed-defect-corpus-current-main.md
  • docs/doctoring/noema-repair-attempt-telemetry.md
  • docs/product-technical-gap-baseline.md
  • scripts/ci/noema_review_gate.py
  • tests/conftest.py
  • tests/test_noema_class_evidence_observation_contract.py
  • tests/test_noema_observed_defect_corpus_current_main.py
  • tests/test_noema_repair_attempt_telemetry.py
  • tests/test_noema_review_gate.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Contributor Author

Coordination note, no action needed from anyone right now: this PR and #1879 (also open) both edit scripts/ci/noema_review_gate.py's _extract_http_error_telemetry/call_llm path, and both touch the exact same test function in tests/test_noema_review_gate.py (test_call_llm_reports_only_safe_model_from_bounded_http_error) — this adds the failure_kind telemetry field there, #1879 wraps the same call site's exception handling to close the HTTP response body and asserts on it in that same test. They don't look semantically conflicting (different concerns: new telemetry field vs. resource cleanup), but a textual merge conflict on that one test function is likely whichever lands second. Flagging so whoever restacks second does a real merge rather than being surprised by it.


Generated by Claude Code

seonghobae and others added 2 commits September 5, 2026 19:48
Preserve PR #1898 runtime behavior and extend the existing HTTP error test across malformed values and length/control-character boundaries. Both the public annotation and raised diagnostic must obey the same field contract.

The exact original head passed 123 tests with hash-locked tooling. Removing only failure-kind extraction/formatting reproduced 3 failures and 13 passing rejection cases; restoring it passed all 138 focused tests. No provider call, review approval, PR closure, or gate change is claimed.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Seongho Bae <me@seonghobae.me>
@opencode-agent
opencode-agent Bot disabled auto-merge September 5, 2026 10:52
seonghobae and others added 7 commits September 5, 2026 20:02
Keep the existing failure_kind delta. Preserve bounded error.code from the protected gateway envelope in both failed-call diagnostics; do not infer failure cause from the HTTP exception label. The expanded field regression reproduced four failures before the fix; focused Noema and declared-pip environment regression: 159 passed.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Address the independent sparse-envelope finding while retaining the existing error telemetry implementation. Removing code extraction reproduced 8 failed/43 passed; restored field matrix 51 passed and focused Noema/edge/environment tests 176 passed. Remove one unused test import found by Ruff. Record the protected-source contract, unknown historical 502 cause, logging limits, separate verification stages, and APA logging guidance in the existing doctoring and gap baseline.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Seongho Bae <me@seonghobae.me>
Non-force merge current protected main into #1641. Reject model-authored runtime, command-output, toolchain-help, and authoritative external-source claims unless a typed out-of-band receipt is explicitly cited. Preserve source-only reasoning, verification directions, exact-line evidence, and fail-closed findings.

Grounded by ConceptWeave #35 review 5120903874 and Noema run 33938445009/job 101256294197, which executed no Cargo or documentation lookup. Removed both purpose-complete temporary writer workflows.

Validation: 268 Noema tests; full 2,922 passed, 1 skipped, 21 subtests.
Resolve the remaining valid #1641 review findings. One unified-diff state machine now emits both exact source text and accepted coordinates. Bounded truncation drops the incomplete final line instead of manufacturing +/- source, so a genuine line equal to the historical omission marker remains reviewable.

Focused RED reproduced both failures before the repair. GREEN: focused 3 passed; Noema 268 passed; full 2,922 passed, 1 skipped, 21 subtests; py_compile and diff checks clean.

Copy link
Copy Markdown
Contributor Author

Dependency-order restack (2026-09-05 UTC)

#1641 and #1898 both modified the canonical Noema validator from protected main@f250638827f8252b0d9e5cb2601f4d333f96162f and had diverged. #1641 is now the dependency root; this later telemetry PR is its ordinary non-force descendant.

The provenance receipt boundary, unified diff parser, whitespace-only evidence guard, and bounded error.code/failure_kind telemetry all coexist on the candidate. No predecessor hosted check/review evidence transfers to this head.

Copy link
Copy Markdown
Contributor Author

Writer handoff — source writer released

The bounded source integration and local verification for this PR are complete. There is no active #1898 source-writing process or verification command in this automation.

  • exact head: a7afb39d8c9e6b1b20857e378a236f566c94ff36
  • tree: fdf57270e965050f7c781ef40303af48d5e6dd3c
  • ordinary merge parents: 0db01c2615457430018a584be1394f57dfdd7038 and evidence-provenance root 294601776987407b2feac45eb949656dd579d655
  • completed local evidence: Noema 319 passed; repository 2,973 passed · 1 skipped · 21 subtests; compilation/diff checks clean

Those results belong to this local execution only. They do not transfer predecessor coverage or replace fresh exact-head hosted checks/review. The originating read-only audit may safely continue without treating this automation as an active competing writer. No source, rerun, model call, approval, or merge was performed in this handoff.

Copy link
Copy Markdown
Contributor Author

Ordinary non-force integration of the updated #1641 dependency root is complete.

Exact #1898 head: 46a01788c181e1228889ed96e792253f82a3c619
Parents:

  1. predecessor fix(noema): expose structured failure kind #1898 a7afb39d8c9e6b1b20857e378a236f566c94ff36
  2. current fix(noema): enforce observed defect-class review evidence #1641 9df1ea4c03521aa69e9ba6b48fa4d940d858fb0c
    Combined tree: 86383910a4b422e1eed69da98b8f80fe3b96c65c

Fresh checkout GREEN on that exact combined head:

  • Noema-related suite: 586 passed
  • full normal: 2976 passed, 1 skipped, 21 subtests
  • full GITHUB_ACTIONS=true: 2976 passed, 1 skipped, 21 subtests
  • full branch coverage: 2976 passed, 1 skipped, 21 subtests; scripts/ci/noema_review_gate.py 953 statements / 444 branches, 100%

This integrates strict outbound-schema/local-validator parity and the corrected invalid-location telemetry fixture without transferring the predecessor 0db01c coverage claim. No workflow rerun, model call, review dismissal, approval, merge, or bypass was issued. #1898 source/verification writer is released at 46a01788c.

Copy link
Copy Markdown
Contributor Author

Protected main adoption is now included through the current #1641 dependency root.

Current #1898 exact head: f1cd87504ec773dac17027b90857cf139111abcc
Parents: prior #1898 46a01788c181e1228889ed96e792253f82a3c619 + current #1641 409638a918e9a224534f7b8779c63cc15ddda801
Tree: 51d7a438ee329708f7304518c83b90c8cb64320e

Fresh exact-head results:

  • full normal: 2977 passed, 1 skipped, 21 subtests
  • full GITHUB_ACTIONS=true: 2977 passed, 1 skipped, 21 subtests
  • branch coverage: 953 statements / 444 branches in noema_review_gate.py, 100%

No predecessor GREEN, historical review, or prior coverage was transferred. No rerun, model call, self-approval, bypass, or protected merge was issued. #1898 source/verification writer is released at f1cd8750.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant