Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
d2094b5
test(packaging): require fast-mlsirm on supported runtimes
seonghobae Sep 1, 2026
3718b7b
fix(packaging): require fast-mlsirm on supported runtimes
seonghobae Sep 1, 2026
e83d6d0
build: refresh Python 3.12 runtime locks
seonghobae Sep 1, 2026
972e055
Merge remote-tracking branch 'origin/main' into fix/require_fast_mlsi…
claude Sep 2, 2026
91f25f9
Merge branch 'main' into fix/require_fast_mlsirm_runtime
opencode-agent[bot] Sep 2, 2026
ecac722
test(release): RED — extraction helper for release notes does not exi…
claude Sep 2, 2026
b7ac64d
feat(release): GREEN — extract release notes from CHANGELOG.md and py…
claude Sep 2, 2026
c9b0154
test(release): RED — release workflow does not exist yet
claude Sep 2, 2026
26e61a6
feat(release): GREEN — canonical immutable release workflow
claude Sep 2, 2026
ad73237
docs(adr): 0127 — canonical immutable release, gated on protected-mai…
claude Sep 2, 2026
a827880
docs: add RELEASING.md maintainer runbook
claude Sep 2, 2026
27136fe
docs(gaps): record the canonical immutable release mechanism landing
claude Sep 2, 2026
615cb55
docs(adr): renumber canonical immutable release ADR 0127 -> 0129
claude Sep 2, 2026
eea42bd
fix(release): idempotent publish + least-privilege split + final tip …
claude Sep 2, 2026
00fe294
fix(release-notes): bound version parsing to [project] table
claude Sep 2, 2026
a869042
docs(release): fix /releases/latest mutability claim + add ADR grounding
claude Sep 2, 2026
cf69dc3
fix(release): checks-green gate, resumable asset attach, document res…
claude Sep 2, 2026
b7a044d
fix(release): checks-registration race and confirmed-absence lookups
claude Sep 2, 2026
8a777a1
fix(release): tag-only resume evaluates the tag's own commit, not mai…
claude Sep 2, 2026
9b93a21
test(release): fail closed on tag namespace and missing SBOM
seonghobae Sep 2, 2026
788dfce
fix(release): bind exact tag ref and require SBOM evidence
seonghobae Sep 2, 2026
29ee4ce
test(release): assert repaired mandatory supply-chain steps
seonghobae Sep 2, 2026
d22586f
docs(release): make tag identity and SBOM gate code-current
seonghobae Sep 2, 2026
a304d74
docs(adr): make immutable-release decision code-current
seonghobae Sep 2, 2026
daa38ec
Merge branch 'main' into fix/require_fast_mlsirm_runtime
opencode-agent[bot] Sep 3, 2026
c4aacc9
Merge branch 'main' into fix/require_fast_mlsirm_runtime
opencode-agent[bot] Sep 3, 2026
5c7ce27
Merge branch 'main' into fix/require_fast_mlsirm_runtime
opencode-agent[bot] Sep 4, 2026
e2df780
Merge branch 'main' into fix/require_fast_mlsirm_runtime
opencode-agent[bot] Sep 4, 2026
9928e4a
fix(nim): refresh hosted access evidence
seonghobae Sep 4, 2026
efd8032
docs: track NIM evidence refresh
seonghobae Sep 4, 2026
28a887e
merge(stack): inherit required fast-mlsirm runtime
seonghobae Sep 4, 2026
712da76
merge(stack): inherit current NIM cost evidence
seonghobae Sep 4, 2026
f753f45
test(release): align immutable publication contracts
seonghobae Sep 4, 2026
6c25848
Merge origin/main; keep this branch's NIM citation and its validator
claude Sep 5, 2026
34cf7de
fix(release): align required checks with integrated quality workflow
seonghobae Sep 7, 2026
92ca4cf
fix(release): verify published SBOM bytes without overwriting assets
seonghobae Sep 7, 2026
8443719
test(release): reject missing or empty downloaded SBOM
seonghobae Sep 7, 2026
27293e6
docs(release): distinguish current publication evidence from history
seonghobae Sep 7, 2026
b0e3d62
Merge remote-tracking branch 'origin/main' into codex/release-check-i…
seonghobae Sep 8, 2026
7a37d59
fix: reject lightweight release tags before resume
seonghobae Sep 12, 2026
b73b5fe
Merge pull request #1147 from ContextualWisdomLab/fix/release-annotat…
seonghobae Sep 13, 2026
1e7f198
docs(release): align ADR with annotated-tag contract
seonghobae Sep 13, 2026
b51009c
docs(release): reject lightweight-tag resume in runbook
seonghobae Sep 13, 2026
1f218cb
test(release): reproduce immutable publication and resume failures
seonghobae Sep 13, 2026
fdc6d4a
fix(release): verify draft assets before immutable publication
seonghobae Sep 13, 2026
c525ae4
docs(release): distinguish verified immutable inventory from release …
seonghobae Sep 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
456 changes: 456 additions & 0 deletions .github/workflows/release.yml

Large diffs are not rendered by default.

48 changes: 48 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,50 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)

## [0.2.0] - Unreleased

### Added

- A canonical, immutable release mechanism: `.github/workflows/release.yml`
(`workflow_dispatch` only, explicit `version` input, never triggered by
push/schedule/merge), split into a read-only, credential-less `verify` job
and a write-scoped `publish` job for least privilege. `verify` checks the
dispatched commit is protected `main`'s untampered current tip and that
every one of this repository's own known push-triggered checks (Tests,
Fuzz, Security's jobs — `RELEASE_EXPECTED_PUSH_CHECKS`) has actually
registered as a check-run for that exact commit *and* every check GitHub
reports for it is complete with an acceptable conclusion (excluding this
release run's own checks) — a dispatch fired moments after a merge, before
GitHub has finished registering those push-triggered check-runs at all,
is correctly "not ready" rather than a vacuous pass on an empty report —
checks the requested version against `pyproject.toml`'s `[project]` table
(table-boundary aware, so a same-named `version` key in an unrelated table
can never be mistaken for it), resolves any existing `vX.Y.Z` tag via the
GitHub commits API (rejecting only one that points at a different commit;
a tag at this commit is always a safe idempotent resume, whether or not
its Release already exists — see below; a failed tag or Release lookup is
read as "absent" only on a confirmed 404 / "release not found" — any other
lookup failure, e.g. a rate limit or transient network/5xx error, fails
the step closed instead of guessing, so a later dispatch retries and
resolves cleanly rather than compounding a wrong assumption), re-runs the
full test suite fresh, renders release notes from this file's matching
`## [X.Y.Z]` section via the tested `scripts/ci/release_notes.py`, and
best-effort looks up a CycloneDX SBOM (a missing SBOM or failed lookup
warns, never blocks). `publish` re-verifies `main`'s tip has not advanced
and every expected check is still registered and green since `verify`
started testing — immediately before it creates anything — then creates
the annotated `vX.Y.Z` tag (skipped on a tag resume) and the GitHub
Release (skipped on a Release resume, e.g. a prior run whose asset upload
failed after the Release itself was already created), always attempting
the best-effort SBOM asset attach afterward either way. Gives downstream
consumers (`ContextualWisdomLab/keyverse#132`, `bandscope#881`, and the
Wardnet consumer-owner handoff, all recorded on
`contextual-orchestrator#971`) an immutable pin target
(`.../releases/tag/vX.Y.Z` — not the mutable `.../releases/latest` alias)
instead of a vendored source SHA. See
`docs/planning/adrs/0129-canonical-immutable-release.md` and
`docs/RELEASING.md`. No release has been cut yet — landing this mechanism
and dispatching the first `v0.2.0` release are deliberately separate
actions.

### Deprecated

- Internal callers now use
Expand All @@ -20,6 +64,10 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)

### Fixed

- Refresh the time-bounded NVIDIA hosted NIM access evidence against the
official Run NIM Anywhere terms, restoring fail-closed live benchmark
execution through 2026-10-04 without treating prototype access as production
pricing or licensing evidence.
- Workflow workers now preserve the caller message array exactly once, while
the added envelope carries only the subtask and Conductor-style prior-step
access list instead of duplicating the task or source attachments.
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -419,6 +419,8 @@ python tests/test_commercial_evidence_export.py
python tests/test_commercial_acceptance_check.py
python tests/test_release_authorization.py
python tests/test_release_authority_snapshot.py
python tests/test_release_notes.py
python tests/test_release_workflow_contract.py
python tests/test_commercial_buyer_acceptance_workflow.py
python tests/test_commercial_release_candidate.py
python tests/test_commercial_gap_register.py
Expand Down
2 changes: 1 addition & 1 deletion conductor/tech-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Language

Python 3.11+.
Python 3.12+.

## Dependencies

Expand Down
1 change: 1 addition & 0 deletions conductor/tracks.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@
| 002-enterprise-design-foundation | active | Add paper-grounded screen design, user stories, REST API, code/DB conventions, and i18n |
| 003-reasoning-effort-profiles | active | Issue #568: versioned per-role `reasoning_effort_profile`, equal-budget θ̂ RMSE ablation, snapshot on run/stream/batch, production defaults locked |
| 004-nim-cost-quality-benchmark | active | Evidence-grade NIM catalog discovery, all-modality capability probes, and the route/conduct/single-worker cost-quality benchmark (docs/nim_benchmark.md) |
| 005-canonical-immutable-release | active | Deliberate `workflow_dispatch` release mechanism (ADR 0129): protected-main-tip + version + fresh-suite gate, CHANGELOG-derived notes, annotated tag, GitHub Release — so consumers pin a release instead of vendoring source (docs/RELEASING.md) |
10 changes: 5 additions & 5 deletions contextual_orchestrator/nim_benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,10 +112,10 @@ def estimate_tokens(text: str) -> int:

ACTUAL_COST_EVIDENCE: dict[str, Any] = {
"evidence_schema_version": "1.0.0",
"source_title": "NVIDIA NIM General FAQ",
"source_url": "https://docs.api.nvidia.com/nim/docs/product",
"source_title": "Run NIM Anywhere",
"source_url": "https://docs.api.nvidia.com/nim/docs/run-anywhere",
"reviewed_at_date": "2026-09-05",
"valid_until_date": "2026-10-05",
"valid_until_date": "2026-10-04",
"access_program": "NVIDIA Developer Program API Catalog hosted endpoints",
"access_scope": "free API endpoint access for prototyping",
"production_access_note": (
Expand Down Expand Up @@ -2535,9 +2535,9 @@ def _validate_actual_cost_evidence(report: dict[str, Any]) -> None:
raise BenchmarkContractError(
"actual cost evidence must preserve the reviewed zero-cost value"
)
if evidence["source_url"] != "https://docs.api.nvidia.com/nim/docs/product":
if evidence["source_url"] != "https://docs.api.nvidia.com/nim/docs/run-anywhere":
raise BenchmarkContractError(
"actual cost evidence must cite the reviewed NVIDIA NIM General FAQ"
"actual cost evidence must cite the reviewed NVIDIA NIM access terms"
)
reviewed_at = _parse_evidence_date(evidence["reviewed_at_date"], "reviewed_at_date")
valid_until = _parse_evidence_date(evidence["valid_until_date"], "valid_until_date")
Expand Down
164 changes: 164 additions & 0 deletions docs/RELEASING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
# Releasing

This document is for a human maintainer cutting a real, immutable
`contextual-orchestrator` release. It is deliberately narrow — see
[`docs/planning/adrs/0129-canonical-immutable-release.md`](planning/adrs/0129-canonical-immutable-release.md)
for the full design and its explicit non-goals.

## What a release is, and is not

A canonical release is an annotated git tag `vX.Y.Z`, a published GitHub
Release with `immutable: true`, and its verified mandatory SBOM. A versioned
URL alone is not evidence that GitHub has locked the tag and assets.

- `.../releases/tag/vX.Y.Z` identifies the version-specific release. Consumers
must verify its immutable state, exact commit, and asset attestation before
admitting that version to a released-owner inventory.
- `.../releases/latest` is a **mutable discovery alias**, not a consumer pin.

A release is **not** the same thing as
[`/api/v1/commercial_release_candidates/latest`](commercial_release_candidate.md)
or `contextual_orchestrator/release_authorization.py`. Those evaluate
commercial/buyer readiness inside the running gateway. This release mechanism
only establishes a citable, verified release for an exact protected commit.
It does not establish that every consumer-required API, authentication,
deployment-identity or transport contract is implemented by that release.
LifeOS and other consumers must verify their specific owner contracts too.

## Preconditions

1. The version to release is already merged to `main`: `pyproject.toml`'s
`[project].version` is the exact `X.Y.Z`, integrated through normal review
and required checks without exceptions.
2. `CHANGELOG.md` has a non-empty `## [X.Y.Z]` release section. An
`- Unreleased` or dated suffix is supported.
3. Either `refs/tags/vX.Y.Z` does not exist, or it is an **annotated** tag
that resolves directly to a commit in protected `main` history. A resume
uses that tag's exact target, not the current dispatch commit. Lightweight
tags, unsupported tag objects and unrelated histories fail closed. Never
replace, promote, move or reuse a tag to make publication succeed.
4. The exact commit has passing required checks. The workflow rechecks
registered push-triggered jobs and the reported check rollup through
`scripts/ci/release_checks_gate.sh`, and runs the full test suite fresh.
Its existing check policy accepts terminal success, skipped or neutral
conclusions; this is not permission to treat a skipped required semantic
security action as actual review evidence. Protected integration still
requires all applicable organization gates and reviews.
A newly merged commit whose expected push checks have not yet registered
is not ready. Re-dispatch after the genuine required evidence exists;
do not weaken the expected check inventory.
5. A successful `security.yml` run for that exact commit exposes a non-empty
`cyclonedx-sbom/cyclonedx-sbom.json` artifact. Lookup, download, upload,
empty-file or content-verification failure is fatal, not best effort.
6. A repository administrator has enabled GitHub release immutability before
publication. The normal workflow token has no Administration permission;
do not add an administrative secret or expand the publisher's authority
merely to read or change this setting. The publisher validates the actual
public `immutable: true` result and release/asset attestations before
reporting success. A setting that was disabled or changed during publication
can leave a complete but mutable public release; that is a **failed** run
and is ineligible for consumption, not an automatic deletion/retagging case.
7. The runner's GitHub CLI supports `gh release verify` and
`gh release verify-asset`. Missing verification capability fails closed;
do not replace it with a filename or hash-only success claim.

## Cutting a release

1. Use **Actions → Release → Run workflow**, or
`gh workflow run release.yml -f version=X.Y.Z`.
2. Select branch `main`; other refs cannot publish.
3. Enter `X.Y.Z` without a leading `v`, matching `[project].version` exactly.
4. The read-only `verify` job:
- classifies the exact `refs/tags/vX.Y.Z`, rejecting lightweight tags;
- peels an annotated tag to its commit and verifies protected-main ancestry;
- treats only confirmed 404 / release-not-found lookups as absence, not
rate-limit, authentication, network or server errors;
- checks out `TARGET_SHA`; a fresh publication must match the current
protected-main tip, while a resume may use its verified ancestor;
- checks the exact-head rollup and project version;
- runs `uv run --locked --extra api --extra db --extra queue --group dev
python -m pytest -q`;
- renders notes from the exact commit's CHANGELOG section;
- downloads the exact-commit mandatory CycloneDX SBOM and passes both
notes and SBOM to the publisher through an Actions artifact.
5. The write-scoped `publish` job, only after verification succeeds:
- rechecks fresh-main identity and exact-target checks before mutation;
- requires non-empty notes and SBOM inputs;
- creates and pushes an annotated tag only for a fresh publication;
- verifies the exact remote tag object and its peeled target, so GitHub
cannot synthesize a tag from a default branch;
- admits an existing release only as a typed, matching-tag, non-prerelease
Draft, or a complete already-published immutable release;
- creates a new release with `--verify-tag --draft`;
- attaches any missing SBOM **only while the release is a Draft**, then
downloads it and compares its bytes with the verified input;
- publishes the verified Draft. An already-public immutable release is
verify-only and is neither recreated, edited nor uploaded to;
- requires the resulting release to be non-Draft, non-prerelease,
matching-tag and `immutable: true`, then runs `gh release verify` and
`gh release verify-asset` for the exact SBOM.
6. Confirm the version-specific release, successful publication run, exact
tag/commit, immutable state, SBOM and its signed attestation. Do not admit
a release merely because it appears in the GitHub Releases list.

## Recovery and known limitations

An interrupted Draft is recoverable without deleting it or moving its tag.
The same version can resume missing-asset upload or final publication after
all current gates pass. Existing assets are compared byte-for-byte and are
never clobbered. A public release missing its mandatory asset, a mutable
public release, mismatching tag, prerelease, untyped lifecycle field or
unavailable metadata is rejected before resume-side mutation.

A completed immutable release can be checked again without mutation. An
attestation failure, including a temporarily unavailable attestation, is
non-passing evidence; a later verification retry does not require deleting
or republishing that release.

The final main-tip check and tag creation are not an atomic compare-and-set.
A concurrent merge can land in that interval. The publisher checks exact tag
identity, but a verified fresh tip might no longer be the newest tip by the
time publication completes. Correct the version through another normal
release; never move the previous tag. For an annotated-tag resume, all gates
bind to that tag's target rather than the newer main tip.

Draft assets remain mutable until publication, so another authorized publisher
can race the draft content comparison. The post-publication signed asset
verification detects a mismatch and fails the run; it does not roll back
publication or authorize a mismatching asset for consumers. Repository-wide
publisher serialization and maintainer access control remain necessary.

Release immutability must be enabled by the repository owner. This workflow
neither changes that administrative setting nor treats a successfully created
but mutable public release as safe. Enabling the setting afterward does not
retroactively validate a failed release. Investigate it and use a new reviewed
version rather than automatically deleting or recycling its tag.

## After a release

Bump `[project].version` and open the next non-empty CHANGELOG section in an
ordinary PR. Consumers with owner handoffs, including `keyverse#132`,
`bandscope#881`, `contextual-orchestrator#971` and LifeOS's `#1023` dependency,
can propose a version bump only after their required released API and runtime
contracts have been verified. A tag-only, source-checkout or unreleased branch
substitution is not a consumer repair.

## Rollback

Do not delete, retag or overwrite a published release to repair a defect.
Publish a new patch/minor version through the same reviewed process. Deleting
an immutable release is not a way to reuse its tag name. Incomplete Drafts
are resumed rather than deleted by this workflow.

## Normative references

- [GitHub immutable releases](https://docs.github.com/en/code-security/concepts/supply-chain-security/immutable-releases)
- [GitHub CLI release creation](https://cli.github.com/manual/gh_release_create)
- [Release attestation verification](https://cli.github.com/manual/gh_release_verify)
- [Release asset verification](https://cli.github.com/manual/gh_release_verify-asset)

The executable local regression is
`python -m pytest -q tests/test_release_immutable_publication.py`. It executes
the publisher's real shell against a stateful CLI boundary with no network or
publication authority. This is scoped regression evidence, not a substitute
for hosted exact-head checks, independent review, or real release verification.
9 changes: 5 additions & 4 deletions docs/doctoring/nim-benchmark-evidence-grade.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,14 +127,14 @@ issuing five calls.
Actual access cost and hypothetical production cost are separate fields and
separate evidence classes.

As reviewed on 2026-09-05, NVIDIA's NIM General FAQ states that NVIDIA Developer
As reviewed on 2026-09-05, NVIDIA's Run NIM Anywhere page states that NVIDIA Developer
Program members have free access to hosted NIM API endpoints for prototyping.
The same source distinguishes development, testing, research, and evaluation
from production and states that production requires NVIDIA AI Enterprise. The
report therefore records `actual_cost_usd = 0.0` only for the reviewed hosted
endpoint access context, includes the exact source, review date, validity
horizon, program scope, production distinction, and uncertainty, and refuses a
live run after 2026-10-05 until the source is reviewed again.
live run after 2026-10-04 until the source is reviewed again.

No NVIDIA model price is embedded or inferred. A live hypothetical pricing
scenario is optional; absence means `unknown`. If supplied, it must be marked
Expand Down Expand Up @@ -237,8 +237,9 @@ Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., … Kore
Y. (2023). Holistic evaluation of language models. *Transactions on Machine
Learning Research*. https://doi.org/10.48550/arXiv.2211.09110

NVIDIA Corporation. (n.d.). *General FAQ*. NVIDIA NIM Documentation. Retrieved
August 5, 2026, from https://docs.api.nvidia.com/nim/docs/product
NVIDIA Corporation. (n.d.). *Run NIM Anywhere*. NVIDIA NIM Documentation.
Retrieved September 5, 2026, from
https://docs.api.nvidia.com/nim/docs/run-anywhere

NVIDIA Corporation. (2026, June 4). *NIM offerings*. NVIDIA NIM for Large
Language Models. https://docs.nvidia.com/nim/large-language-models/2.0.5/about-nim-llm/nim-offerings.html
Expand Down
4 changes: 2 additions & 2 deletions docs/nim_benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,11 +150,11 @@ actual and hypothetical cost fields, and a response SHA-256.
Actual endpoint access and hypothetical paid cost remain separate evidence
classes.

As reviewed on 2026-09-05, NVIDIA's current General FAQ states that NVIDIA
As reviewed on 2026-09-05, NVIDIA's Run NIM Anywhere page states that NVIDIA
Developer Program members have free access to hosted NIM API endpoints for
prototyping. The report records that exact source, review date, validity horizon,
program context, production distinction, and uncertainty. A live run fails
closed after 2026-10-05 until the official source is reviewed again. Production
closed after 2026-10-04 until the official source is reviewed again. Production
support and licensing are not inferred from prototype access and require
NVIDIA AI Enterprise under the reviewed documentation.

Expand Down
Loading
Loading