feat(reviewer): bind evidence receipts to exact claims - #556
Conversation
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (16)
📝 WalkthroughWalkthroughclaim-evidence 영수증과 canonical manifest 검증을 추가했다. 중앙 workflow는 manifest checksum과 Sigstore attestation을 생성한다. CLI와 리뷰 agent는 검증된 receipt만 prompt와 finding publication에 사용한다. source 및 sandboxed_verify producer와 관련 테스트도 추가했다. ChangesClaim-evidence 검증 및 게시
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant CentralReview
participant ClaimEvidenceRuntime
participant Sigstore
participant ReviewerCLI
participant ReviewAgent
CentralReview->>ClaimEvidenceRuntime: 생성된 source claim manifest
CentralReview->>Sigstore: manifest attestation 생성
Sigstore-->>CentralReview: attestation bundle
CentralReview->>ReviewerCLI: manifest와 실행 identity 전달
ReviewerCLI->>Sigstore: attestation 검증
ReviewerCLI->>ClaimEvidenceRuntime: manifest index 검증
ReviewerCLI->>ReviewAgent: 검증된 claim-evidence index 바인딩
ReviewAgent->>ClaimEvidenceRuntime: model verdict evidence admission
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
seonghobae
left a comment
seonghobae
left a comment
There was a problem hiding this comment.
Exact-head owner review for 2571d7f996161df32227a27fc66b502bbb1b5d31.
Blocking trust-boundary finding: admit_claim_evidence() validates a caller-supplied model/Pydantic payload and binds only shared identity, required kind, claim digest, artifact digest, and artifact size. It does not authenticate a producer envelope or bind the variant's semantic evidence fields to the exact artifact. With the same admitted claim/artifact bytes, an attacker can change ExecutionClaimReceipt.argv, exit_code, stdout_sha256, stderr_sha256, isolation_policy, or network_policy; likewise SourceClaimReceipt.source_path/source_line/source_line_sha256 and ResearchClaimReceipt.source_uri/source_revision/excerpt_sha256/retrieval_policy can be substituted while admission still succeeds. The current hostile tests mutate only shared identity/digest fields and therefore do not expose this.
The docstring and PR claim a “sealed/producer-issued” receipt, but the API accepts an arbitrary payload without verifying a trusted producer signature/MAC or looking up a trusted immutable manifest. Freezing a Pydantic model does not establish provenance. Add RED cases that keep the exact artifact fixed while mutating every meaning-bearing variant field, then make the publication boundary resolve a model-cited receipt ID from a trusted producer manifest/index (or verify an authenticated envelope that covers the complete canonical payload). The model must not submit the authoritative receipt dict. Extend the real producer → gate → finding/publication path with positive/negative integration tests; a standalone schema module alone does not close .github#1641 / Concept35.
This is an owner-side contract review, not approval, release evidence, or permission for the central consumer to proceed. Preserve the existing identity/digest hardening; the missing delta is complete producer-envelope binding and actual publication-boundary integration.
seonghobae
added
enhancement
ChatGPT Codex Connector
|
Read-only review follow-up on exact head
This is not completion of issue #555 or the Concept35 consumer finding. Existing |
Advance the documentation contract to the live #556 successor after its hosted release-test RED, while rejecting the superseded observation. Production baseline text follows in the causal repair commit.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
Fresh integration sweep: unchanged exact |
* test(docs): expose post-547 commercial authority drift * test(docs): require current active commercial lanes * docs: refresh commercial gap authority after #547 * test: advance patch-validator candidate authority * test: reject superseded patch-validator candidate * docs: refresh patch-validator exact authority * test: require current #535 commercial authority * docs: repair current commercial authority after hosted RED * test(docs): require live post-558 commercial authority Require the documentation lane to reflect protected #558 integration, the newly converged #535 exact head, and the newly observed #556 stacked head before production baseline text is repaired. * docs(gap): repair post-558 live commercial authority Bring #559's owned baseline in line with protected #558 integration, exact #535 convergence, and the newly observed #556 stack while preserving canonical owner boundaries and release-evidence discipline. * test(docs): require latest observed #556 authority Advance the documentation contract to the live #556 successor after its hosted release-test RED, while rejecting the superseded observation. Production baseline text follows in the causal repair commit. * docs(gap): record live #556 successor and hosted RED Update #559's sole documentation authority to the latest #556 exact head, preserve the observed hosted release-test failure as historical evidence, and keep the downstream stack non-authorizing until #535 reaches protected main. * test(docs): require current #535 endpoint-repair authority * docs: track current #535 gateway endpoint repair * test(docs): require #535 coverage-repair authority * docs: track #535 coverage-gate repair * test(docs): require current central control-plane snapshot * docs: refresh central control-plane snapshot * test(docs): retire stale commercial authority assertions * docs: refresh protected and claim-evidence authority * test(docs): track protected #535 integration * test(docs): track current claim-evidence head * test(docs): bind post-535 protected authority * test(docs): refresh current candidate contract * test(docs): align claim-binding authority wording * docs: refresh claim-evidence candidate authority * test(docs): match hosted run authority casing * docs: refresh live Noema commercial authority * test(docs): track current Noema feature authority * test(docs): bind commercial gap to live feature heads * test(docs): require settled external-extension authority * docs: converge commercial gap to current plugin admission * test(docs): track active policy-approval RED * docs: bind commercial gap to policy-approval RED * docs: restore durable commercial gap owners * docs: align external-extension authority after restack * test: bind documentation authority to restacked #560 * test(d(docs): require complete gap authority schema * docs: bind gap status to architecture authorities * test(docs): reject stale tool-capability candidate * docs: refresh current tool-capability evidence * docs: refresh active extension authority * docs: bind extension event chronology evidence * docs: refresh extension chronology authority * docs: bind gap baseline to hostile admission repair * test(docs): bind live candidate to admission repair * test(docs): preserve candidate ADR authority wording * docs: refresh external-extension runtime-time authority * test(docs): bind current runtime-time candidate authority * test(docs): refresh live external-extension candidate authority * docs: bind gap authority to invocation replay repair * docs: converge on public replay authority repair * docs: converge on activation revocation repair * docs: converge on admission-bound invocation authority * docs: bind commercial gap to exact admission provenance * test(docs): require exact-admission candidate authority * test(docs): bind candidate contract to exact admission * docs: track crypto-provider RED authority * test(docs): bind external-extension authority to Web Crypto repair * docs: converge external-extension Web Crypto authority * test(docs): follow current external-extension authority * test(docs): assert public receipt binding authority * test(docs): match activation revocation authority wording * docs: converge baseline after #560 protected integration * test: bind documentation authority to #560 integration * test: treat external-extension admission as protected history * test: extend protected integration history through #560 * test: move external-extension gap to lifecycle successor
1 participant
Scope and ownership
Owned by issue #555. This remains the canonical Draft successor for the Concept35 provenance defect. Noema owns claim-evidence admission and reviewer publication; provider routing remains
contextual-orchestratorauthority, while isolation, research retrieval, security workflow and outbound authority remain with their canonical owners.Current authority — 2026-09-08 KST
Prerequisite #535 normally integrated as GitHub-verified protected
main@0dbfceb850cda3a016ceb39ae1c8a1a96a9f2ad5after unchanged exact82b20b293f0a5f0ac0e69857c1b61dddfe478491reached terminal application CI, reviewer-ci, required Security and patch-validator-image success with clean review authority. #556 is ahead-only from that protected base withbehind_by=0and merge-base exactly protected main.Fresh source-authority review found that exported
produce_source_claim_receipt()had accepted a paraphrase claim independently of exactsource_line_bytes. Test-onlydbab4cdc150d4973002f8b61173282f7c7542725established the exact-line contract; its hosted generation was cancelled by normalcancel-in-progressbefore test execution, so no hosted RED is claimed for that intermediate head. Production440346ee73e60e87915bd3027a774e24d3ac5124rejects multi-line/non-UTF-8 source bytes and requires claim equality with the decoded exact line. Edge head920eb7be0c3f57c5f149328a08beddc13339a338covers LF/CRLF/unterminated lines, paraphrase mismatch, embedded multi-line bytes and invalid UTF-8.That edge head produced a real hosted reviewer RED: run
34190991526, job101948849348passed exact checkout, noema-core 100% line+branch coverage and 100% docstring gates, then the reviewer suite ended 1 failed / 721 passed at 100% coverage. The stale_source_bundle()fixture still paired the old runtime paraphrase with exact source bytes, so the new production invariant correctly rejected it. Causal repair1cd8db5a94ed420bed8b52be0d3b3354f9fc86abchanged the fixture to the exact source claim while retaining the intended source-vs-execution kind-mismatch rejection.Fresh review then surfaced one valid no-behavior-change finding: unused
claim_evidence_runtimeimport inreviewer/tests/test_claim_evidence_publication_boundary.py.809ccb78bfdf8f9785d4c74ab834159961cce6e9removed only that unused import. The inline review thread is resolved/outdated.A subsequent direct publisher-boundary review found a vacuous-oracle bypass: a model could return finding-free
request_changesorblocked, skip receipt admission entirely, and publish a GitHubREQUEST_CHANGESreview whose body claimed “No blocking findings.” RED6629f07e2c7bd35698331d8966b7cbe35204a808exercises both non-approval states through the real PydanticAI agent→CLI publisher seam. GREEN4c2153702972e8d5c0f077ac79cc68cbf6da4bfbrequires every model non-approval to contain producer-authenticated findings before deterministic gates or publication; deterministic strict/check/security/dependency blockers remain independently generated after admission. Current exact860714cba46dba06260a5dce09d0e9152fcb0a8calso records the invariant in CHANGELOG.Security/ruleset boundary
Protected
AGENTS.mdremains authoritative: Noema Security ruleset18794436targets~DEFAULT_BRANCH. Feature-base stacked PRs may legitimately lack Security until retargeted tomain; after this PR was retargeted, required Security materialized as intended..github#2037is the narrower central stacked-OpenCode source/live authority issue and is not a request to broaden Noema Security onto feature refs.Claim-level contract
Remote RED
451683fbe38fb3711d151e61249fb9040560eab0reproduced the original agent→CLI publication defect. The retained candidate separates producer receipt semantics from independentClaimEvidenceRequirementpublication authority. Raw current-head source receipts are context only. Source-only findings require exact producer-authenticated claim/evidence kind and coordinates; source receipt production itself requires exact claim-to-line identity. Requirement/receipt mismatch, wrong coordinates, paraphrase substitution, direct model dictionaries and context-to-finding promotion fail closed before deterministic gates and GitHub publication.Current exact-head evidence
Exact
860714c...has a wholly fresh generation; predecessor evidence is non-authorizing. Current application CI34197596549, reviewer-ci34197596588, and required Security Scan34197596536are terminal SUCCESS on the unchanged head. patch-validator-image34197596517, job101968826382, is assigned to a GitHub-hosted runner and remains inBuild exact-head patch-validator image; checkout, stale-head guard, Trivy, pinned Syft/Grype, dependency materialization and Buildx setup already succeeded. This is 3-GREEN, not four-GREEN. Local focused verification at the exact successor tree is 92 passed with 404 statements and 130 branches at 100%, public-doc 100%, compileall andgit diff --checkGREEN. Keep Draft until the image lane reaches terminal success, fresh review authority remains clean and protected ancestry is unchanged.Central
ContextualWisdomLab/.github#1641@b8c986e2406beb37d254acd4c5df6389038b55f2retains the original Concept35 sentence and synonym RED. Completion after source integration still requires producer-authenticated execution stdout/stderr evidence, content-addressed research evidence, an immutable Noema release, then a released central consumer bump and unchanged original/synonym corpus RED→GREEN.No release, consumer GREEN, current-head four-GREEN, self-approval, force update, provider bypass or gate weakening is claimed.
Summary by CodeRabbit
새 기능
문서