Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ Semantic Versioning where the repository publishes a release.

### Fixed

- Retry a Strix provider model that emits its exact model-quality warning via
the configured fallback sequence, while retaining fail-closed handling for
all other warning, timeout, provider, and vulnerability signals.
- Parsed `opencode.jsonc` as JSONC (stripping `//` and `/* */` comments outside string literals) in the reasoning-effort guard and its contract tests, instead of raw `json.loads`, which rejected the file the moment it carried its first explanatory comment (added for the `contextual-orchestrator` provider block) with `Expecting property name enclosed in double quotes`. Comment markers inside string values, such as the `$schema` URL, are left untouched.
- Download the pinned `uv` 0.12.1 exporter from the official GitHub Releases URL instead of `releases.astral.sh`, which now returns HTTP 403 and blocks org-wide OpenCode `coverage-evidence`. The SHA-256 pin is unchanged. The opener may follow one hop onto `release-assets.githubusercontent.com` or `objects.githubusercontent.com` and still rejects every other host, userinfo, non-HTTPS scheme, and nondefault port (ContextualWisdomLab/.github#1109).
- Compared the trusted `uv` executable's post-install `--version` output against the real GitHub Releases build's full string, `uv 0.12.1 (x86_64-unknown-linux-gnu)`, instead of the bare `uv 0.12.1` the prior check required; the genuine release binary always prints the target triple, so every installation was failing the pin check immediately after the archive download itself was fixed (ContextualWisdomLab/.github#1109).
Expand Down
49 changes: 49 additions & 0 deletions docs/doctoring/strix-quality-warning-fallback.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Strix model-quality warning fallback

검토 기준일: **2026-08-21**

## Incident

The central Strix workflow selected the public-repository NVIDIA NIM model
`nvidia/nemotron-3-super-120b-a12b`. Strix completed a scan and produced only a
`LOW` finding, but it also emitted `MODEL QUALITY WARNING` because the selected
model was not a recommended frontier model. The gate correctly classified the
warning as non-clean evidence, then stopped before trying the configured
fallback models. This left an actionable low-severity report indistinguishable
from an unrecoverable provider failure and blocked the target pull request.

## Decision

`scripts/ci/strix_quick_gate.sh` recognizes only Strix's exact model-quality
warning (`MODEL QUALITY WARNING` / `is not a recommended frontier model for
Strix`) as retryable model evidence. It remains an infrastructure/failure
signal, so the scan never passes merely because the warning was seen. The
existing fallback sequence must obtain a clean result or the gate fails closed.

All other `Warn`, `Warning`, `Fatal`, `Denied`, and `Timeout` output remains a
hard failure. A `MEDIUM` or higher vulnerability remains blocking even when a
fallback succeeds; below-threshold findings are handled by the existing
`STRIX_FAIL_ON_MIN_SEVERITY` policy.

## Verification contract

`tests/test_strix_nvidia_nim_not_found_fallback.py` executes the production
classifier against a bounded quality-warning log and asserts that the warning
is wired into `is_model_retryable_error`. The existing shell harness continues
to cover generic warning signals and fallback failure paths.

## Rollback

If a future Strix release changes the warning wording, add the exact new
provider-produced wording to the narrow classifier and its regression test.
Do not remove generic warning failure handling or neutralize the entire warning
class.

## References (APA 7th)

GitHub. (n.d.). *Workflow syntax for GitHub Actions*. GitHub Docs. Retrieved
August 21, 2026, from
https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax

GitHub. (n.d.). *Using workflow run logs*. GitHub Docs. Retrieved August 21,
2026, from https://docs.github.com/en/actions/how-tos/monitor-workflows/use-workflow-run-logs
12 changes: 12 additions & 0 deletions scripts/ci/strix_quick_gate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2944,6 +2944,10 @@ is_llm_token_limit_error() {
# was interrupted or incomplete. Used as a guard to prevent the
# below-threshold override from silently passing an aborted scan.
has_detected_infrastructure_error() {
if is_model_quality_warning; then
return 0
fi
Comment thread
seonghobae marked this conversation as resolved.
Outdated
Comment thread
seonghobae marked this conversation as resolved.
Outdated

if grep -Eiq '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning)([^[:alpha:]]|$)' "$STRIX_LOG"; then
return 0
fi
Expand Down Expand Up @@ -2990,6 +2994,10 @@ has_detected_infrastructure_error() {
return 1
}

is_model_quality_warning() {
grep -Eiq 'MODEL QUALITY WARNING|is not a recommended[[:space:]]+frontier model for Strix' "$STRIX_LOG"
}
Comment thread
seonghobae marked this conversation as resolved.

latest_strix_report_dir() {
local latest=""
local run_dir
Expand Down Expand Up @@ -3818,6 +3826,10 @@ is_hallucinated_source_claim_finding() {
is_model_retryable_error() {
local model="$1"

if is_model_quality_warning; then
return 0
fi
Comment thread
seonghobae marked this conversation as resolved.

if is_vertex_model "$model" && is_vertex_not_found_error; then
return 0
fi
Expand Down
35 changes: 35 additions & 0 deletions tests/test_strix_nvidia_nim_not_found_fallback.py
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,41 @@ def test_not_found_skips_same_model_and_enters_cross_model_fallback(self) -> Non
self.assertIn("is_nvidia_nim_not_found_error", retryable)
self.assertNotIn("is_nvidia_nim_not_found_error", same_model_retry)

def test_model_quality_warning_enters_configured_fallback(self) -> None:
"""Retry a weaker provider model instead of treating its warning as clean evidence."""

gate_source = STRIX_GATE.read_text(encoding="utf-8")
quality_warning = _function_block(
gate_source,
"is_model_quality_warning",
)
retryable = _function_block(gate_source, "is_model_retryable_error")
self.assertIn("is_model_quality_warning", retryable)

with tempfile.TemporaryDirectory(prefix="strix-quality-warning-") as temp_dir:
log_path = Path(temp_dir) / "strix.log"
log_path.write_text(
"MODEL QUALITY WARNING\n"
"'nvidia_nim/example' is not a recommended frontier model for Strix.\n",
encoding="utf-8",
)
script = "\n".join(
(
"set -euo pipefail",
'STRIX_LOG="$1"',
quality_warning,
"is_model_quality_warning",
)
)
completed = subprocess.run(
["bash", "-c", script, "strix-quality-classifier", str(log_path)],
check=False,
capture_output=True,
text=True,
)

self.assertEqual(completed.returncode, 0, completed.stderr)

def test_workflow_uses_available_free_first_nvidia_plan(self) -> None:
"""Prefer a documented hosted NIM and another NIM before GitHub."""

Expand Down
Loading