From 425a466ddfe8610015722d7d5b0257cc445d02c5 Mon Sep 17 00:00:00 2001 From: Christopher Patti Date: Mon, 31 Aug 2026 18:08:06 -0400 Subject: [PATCH 1/3] chore(witan-core): pin omnigraph to the released v0.10.0, off the `edge` tag MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `edge` is force-updated on every push to upstream main, so a pinned digest describes a tarball that stops being downloadable the moment upstream merges. It broke twice on 2026-08-31 alone: `witan-code (code graph)` — the one job deliberately left uncached to report exactly this — was red on #300 and #302 at 11:32Z, went green when #305 refreshed the pin at 14:44Z, and was red again on #308/#309 by 20:13Z after four upstream pushes republished the assets. The refresh held about six hours. A fresh `witan setup` fails the same way, for the same reason. Upstream cut v0.10.0 at 21:29Z tagging a625748c8b, which is the same commit the then-current `edge` build came from, so this pins the same source under a name that cannot move rather than adopting a new one. That also ends the "temporarily on edge for a re-test" note above the version pin: the build under test now has a release. Four upstream commits arrive relative to the pinned ac620eea87. Only one carries operator consequence: #581 upgrades Lance to 11, which upstream says changes English stemming, so an index built by Lance 10 read with the Lance 11 analyzer silently under-returns. Every search()/bm25() query in read.gq sits on such an index. _OMNIGRAPH_INTERNAL_SCHEMA stays 6 and the storage-format gate stays green, so nothing catches this automatically — existing graphs need an explicit, per-branch `omnigraph rebuild-full-text-indexes`. Recorded in full above the digests. Verified against the v0.10.0 binary: witan-core 584, witan-council 1009, witan-code 580 — 2173 passed, no failures. The history block records how to run that honestly, since hermetic.py prepends the real ~/.local/bin and a PATH override alone silently tests the machine's existing binary instead. --- docker/omnigraph-server.Dockerfile | 13 +- docker/witan.Dockerfile | 13 +- .../witan_core/omnigraph_install.py | 161 ++++++++++++++---- 3 files changed, 145 insertions(+), 42 deletions(-) diff --git a/docker/omnigraph-server.Dockerfile b/docker/omnigraph-server.Dockerfile index 8e8f4521..0ef5a567 100644 --- a/docker/omnigraph-server.Dockerfile +++ b/docker/omnigraph-server.Dockerfile @@ -31,13 +31,14 @@ # covered it when it did not, and a partial bump is silent until deploy. ARG OMNIGRAPH_VERSION=0.10.0 -# Upstream tag to fetch from. `edge` is the rolling build of upstream main, -# republished on every push there; a real release is `v${OMNIGRAPH_VERSION}`. +# Upstream tag to fetch from. A real release is `v${OMNIGRAPH_VERSION}`; `edge` +# is the rolling build of upstream main, republished on every push there. # Kept separate because on a moving tag the two differ — see -# witan_core/omnigraph_install.py :: _OMNIGRAPH_RELEASE_TAG. -ARG OMNIGRAPH_RELEASE_TAG=edge -ARG OMNIGRAPH_SHA256_X86_64=6a0fba8842a2071c558abf2c1a399ce5e11d359dff78b6ae6ff3676617f95680 -ARG OMNIGRAPH_SHA256_ARM64=dd40fa4169a89af41cddbdeb8fe441b714438633297e153876b4889ec0af3a86 +# witan_core/omnigraph_install.py :: _OMNIGRAPH_RELEASE_TAG, which also records +# why this is a release tag again and what Lance 11 asks of an existing graph. +ARG OMNIGRAPH_RELEASE_TAG=v0.10.0 +ARG OMNIGRAPH_SHA256_X86_64=05d3ce4ec0ab51a876befd89b643c3e7f2d5489be0398a38cef6fb3a0d257fc1 +ARG OMNIGRAPH_SHA256_ARM64=dd3ac09123a68882454db7e689da4c306c41677826237098df4e76b0f73d8d5e # ── Fetch + checksum-verify the release, extract both binaries ──────────────── FROM debian:trixie-slim@sha256:3a39a0592364683e6bab97937b72cad5a8fa6dcbbee90edb3bb48c7f8e94f258 AS fetch diff --git a/docker/witan.Dockerfile b/docker/witan.Dockerfile index e387435c..12c0dd7d 100644 --- a/docker/witan.Dockerfile +++ b/docker/witan.Dockerfile @@ -23,13 +23,14 @@ ARG PYTHON_VERSION=3.14 # version is an outage. Renovate covers all three; `just check-omnigraph-pins` # is the CI backstop. ARG OMNIGRAPH_VERSION=0.10.0 -# Upstream tag to fetch from. `edge` is the rolling build of upstream main, -# republished on every push there; a real release is `v${OMNIGRAPH_VERSION}`. +# Upstream tag to fetch from. A real release is `v${OMNIGRAPH_VERSION}`; `edge` +# is the rolling build of upstream main, republished on every push there. # Kept separate because on a moving tag the two differ — see -# witan_core/omnigraph_install.py :: _OMNIGRAPH_RELEASE_TAG. -ARG OMNIGRAPH_RELEASE_TAG=edge -ARG OMNIGRAPH_SHA256_X86_64=6a0fba8842a2071c558abf2c1a399ce5e11d359dff78b6ae6ff3676617f95680 -ARG OMNIGRAPH_SHA256_ARM64=dd40fa4169a89af41cddbdeb8fe441b714438633297e153876b4889ec0af3a86 +# witan_core/omnigraph_install.py :: _OMNIGRAPH_RELEASE_TAG, which also records +# why this is a release tag again and what Lance 11 asks of an existing graph. +ARG OMNIGRAPH_RELEASE_TAG=v0.10.0 +ARG OMNIGRAPH_SHA256_X86_64=05d3ce4ec0ab51a876befd89b643c3e7f2d5489be0398a38cef6fb3a0d257fc1 +ARG OMNIGRAPH_SHA256_ARM64=dd3ac09123a68882454db7e689da4c306c41677826237098df4e76b0f73d8d5e # Keep in lockstep with witan-council's version (mcp/servers/witan/pyproject.toml # [project].version / [tool.bumpversion]); it labels the built image. ARG WITAN_VERSION=0.8.0 diff --git a/packages/witan-core/witan_core/omnigraph_install.py b/packages/witan-core/witan_core/omnigraph_install.py index 56f328f5..d028c009 100644 --- a/packages/witan-core/witan_core/omnigraph_install.py +++ b/packages/witan-core/witan_core/omnigraph_install.py @@ -37,41 +37,48 @@ import urllib.request from pathlib import Path -#: ★ TEMPORARILY ON `edge` (0.10.0) FOR A RE-TEST — NOT A DECISION TO ADOPT IT. +#: 0.10.0, from the real ``v0.10.0`` release — no longer the `edge` re-test. #: 0.10.0 was reverted on 2026-08-14 for halving the write ceiling -#: (agent-kit#233). Three witan-side confounds have since been fixed and the -#: measurement is worth repeating; see -#: tk-omnigraph-0-10-0-edge-halved-the-write-ceiling-r-7ba7c2 for the -#: hypothesis and the revert procedure. If you are reading this after the -#: experiment concluded, it should already be back on 0.9.0/v0.9.0 — if it is -#: not, that is the bug. +#: (agent-kit#233); three witan-side confounds were fixed and the measurement +#: repeated on `edge` (tk-omnigraph-0-10-0-edge-halved-the-write-ceiling-r-7ba7c2). +#: Upstream cut ``v0.10.0`` on 2026-08-31T21:29Z, which ended the re-test by +#: giving the build under test an immutable tag: the paragraph that used to +#: stand here said the pin should go back to 0.9.0/v0.9.0 if the experiment had +#: concluded, and pinning the released 0.10.0 is that same instruction answered +#: forwards rather than backwards. The write-ceiling task holds the measurement. _OMNIGRAPH_VERSION = "0.10.0" #: WHICH UPSTREAM TAG THE BINARY IS FETCHED FROM. Normally ``v`` + the version #: above; ``edge`` selects the rolling build of upstream ``main``, which #: ``release-edge.yml`` force-updates and re-publishes on every push there. #: -#: Separate from ``_OMNIGRAPH_VERSION`` because on a moving tag the two genuinely -#: differ: ``edge`` currently ships a binary that reports ``0.10.0``, and there is -#: no ``v0.10.0`` release to download. Collapsing them into one string would -#: either break the URL or break the "already installed, skipping" check, which -#: compares against what ``omnigraph --version`` actually prints. +#: Kept as its own constant even now that it is just ``v`` + the version: the +#: two genuinely diverge on a moving tag, and collapsing them would break +#: either the URL or the "already installed, skipping" check, which compares +#: against what ``omnigraph --version`` actually prints. #: -#: ★ A MOVING TAG WEAKENS THAT SKIP CHECK, and the caveat is the price of using -#: one: two different `edge` builds both report ``0.10.0``, so a machine that -#: installed yesterday's will not re-download today's. +#: ★ OFF THE MOVING TAG AS OF 2026-08-31, AND THAT IS THE POINT OF THIS LINE. +#: ``edge`` is force-updated on every push to upstream main, so a pinned digest +#: describes a build that stops being downloadable the moment upstream merges: +#: it moved twice in the seven hours after the 2026-08-31 refresh (#305), and +#: each move turned `witan-code (code graph)` red on every open PR and left a +#: fresh ``witan setup`` unable to fetch a binary at all. A release tag is +#: immutable, so the digests below stay valid until someone deliberately moves +#: them. #: #: There is no flag or environment override for this — the tag is a property of #: the repo, not of a run, precisely because all three tiers must agree on it #: (``just check-omnigraph-pins``). To be certain which build you are on: #: delete the binary and re-run ``witan setup``, which re-downloads and verifies -#: against the digest pinned below. To move OFF the moving tag, edit this -#: constant to ``v`` and refresh those digests in the same commit. +#: against the digest pinned below. To go BACK to a moving tag, set this to +#: ``edge`` and refresh those digests in the same commit — and expect the red +#: check described above to return with it. #: -#: Renovate manages the VERSION line only (see renovate.json). While this is -#: ``edge`` a bump is not meaningful, so pin a real ``v`` before -#: treating dependency updates here as authoritative. -_OMNIGRAPH_RELEASE_TAG = "edge" +#: Renovate manages the VERSION line only (see renovate.json). Now that this is +#: a real ``v``, a Renovate bump here IS meaningful — it must move this +#: tag, the three digests, and both Dockerfiles together, which is what +#: ``just check-omnigraph-pins`` enforces. +_OMNIGRAPH_RELEASE_TAG = "v0.10.0" #: The on-disk storage format ``_OMNIGRAPH_VERSION`` is expected to read, as #: reported by ``omnigraph version``'s ``internal-schema`` line. 0.8.x reads 4; @@ -122,12 +129,17 @@ #: and confirm it against the tarball you actually downloaded (`sha256sum`) in #: the same sitting — on a moving tag the two assets can be republished a #: minute apart, and a digest read across that gap describes neither build. -#: ★ THE DIGESTS BELOW ARE THE `edge` BUILD OF 2026-08-30T23:09Z (through -#: ac620eea87), NOT v0.9.0's. THAT IS THE ONLY LINE HERE STATING WHICH BUILD -#: IS PINNED — everything after it is the refresh history in order, appended -#: rather than rewritten, and each block describes the build CURRENT AT ITS -#: OWN DATE. Read the last block (2026-08-31) for what is pinned now; read -#: the earlier ones for what was checked and what it cost to learn. +#: ★ THE DIGESTS BELOW ARE THE RELEASED `v0.10.0` (2026-08-31T21:29Z), NOT an +#: `edge` build and not v0.9.0's. THAT IS THE ONLY LINE HERE STATING WHICH +#: BUILD IS PINNED — everything after it is the refresh history in order, +#: appended rather than rewritten, and each block describes the build CURRENT +#: AT ITS OWN DATE. Read the last block for what is pinned now; read the +#: earlier ones for what was checked and what it cost to learn. +#: +#: The "confirm it against the tarball you actually downloaded in the same +#: sitting" instruction above was written for the moving tag and is now belt +#: and braces rather than load-bearing: a release tag cannot be republished +#: under you mid-refresh. It was still done for this one. #: #: ── 2026-08-24, the `edge` build through bb0e3dc8bf ── #: Refreshed from the 2026-08-24T12:50Z triple (972f1666c5) after CI failed @@ -360,15 +372,104 @@ #: linux-x86_64 507a36f385bea073e7f284fe476befbb4cd788b32bfa85d6f4cd5e943b663197 #: linux-arm64 6742a7fcf2761cb5841a38990c38383d7a884da2c65e3e7cc884afbbf2b2d881 #: macos-arm64 69f78c93e661e8ea2b92deafe6330650a0921a003c2099b75b226482a90dc03e +#: +#: ★★ 2026-08-31, THE SECOND ENTRY OF THIS DATE, AND IT LEAVES THE MOVING TAG. +#: Not a refresh: `_OMNIGRAPH_RELEASE_TAG` goes `edge` -> `v0.10.0`. +#: +#: WHY NOW — THE PIN BROKE TWICE IN ONE DAY, and the second time is this entry. +#: `witan-code (code graph)` is uncached precisely to report this, so its +#: history dates the breakage: +#: * 11:32Z — red on agent-kit#300 and #302, against the PRE-#305 pin. +#: * 14:44Z — #305 refreshes to ac620eea87. Green again at 14:45-14:53Z +#: (#298, #138, #304), and still green at 16:36Z (#306) and 18:35Z (#307). +#: * Four commits then land upstream (18:28Z, 19:05Z, 19:36Z, 20:16Z), each +#: force-updating `edge` and republishing its assets. +#: * 20:13Z — red again on #308 and #309, and 21:21Z on #310. PRs still +#: showing green are holding results from before the move, not passing now. +#: So the refresh above held roughly six hours. A fresh `witan setup` fails the +#: same way and for the same reason: the pinned digest names a tarball upstream +#: has already replaced. The block above spent its length on which 29 commits +#: arrived; the recurring cost was never the reading, it was the half-life. +#: +#: Upstream cut `v0.10.0` at 21:29Z tagging +#: a625748c8bf41e21654c48321fa31d295add7621 — EXACTLY the commit the +#: then-current `edge` build was cut from (`compare` reports identical) — so +#: this pins the same source under a name that cannot move. The re-test the +#: header paragraph described is over by virtue of its subject shipping. +#: +#: WHAT ARRIVED since the pinned ac620eea87: 4 commits, 67 files. +#: #581 `feat(storage): upgrade to Lance 11 with safe full-text rebuilding` +#: — the only one carrying operator consequence. See the ★ below. +#: #582 `fix(compiler): reject undeclared variables in property matches` +#: A `$var` used in a match property must now be a declared query +#: parameter, or typecheck fails with T3. This is about QUERY TEXT, not +#: about the params dict a caller passes: an extra key the query does not +#: declare is still accepted and ignored. `queries/*.gq` are clean under +#: it — the suites below exercise every one of them. +#: #585 `release: qualify v0.10 upgrade and isolate FTS test counters` +#: Upstream's own cross-version v0.9->v0.10 suite plus release notes. +#: #586 `test(bench): remove synthetic worker timing race` — tests only. +#: +#: ★ LANCE 11 CHANGES ENGLISH STEMMING, AND THE STORAGE-FORMAT GATE WILL NOT +#: TELL YOU. `_OMNIGRAPH_INTERNAL_SCHEMA` stays 6 and `omnigraph version` on +#: this binary agrees, so `bin/check_omnigraph_format.py` is green — correctly, +#: because the on-disk format did not move. What DID move is the full-text +#: analyzer: an index built by Lance 10 read with the Lance 11 analyzer SILENTLY +#: UNDER-RETURNS (upstream's own regression cites `organism` and `university`). +#: Every `search()`/`bm25()` query in `read.gq` sits on such an index — memory +#: search and the Task/WorkflowProject BM25 search both — so on any graph +#: written before this binary, search quietly gets worse rather than failing. +#: The remedy is explicit and per branch: +#: omnigraph rebuild-full-text-indexes --branch +#: Upstream calls it a controlled cutover, not a rolling upgrade: stop old +#: readers/writers and keep a recoverable backup first. A local store is one +#: command; the DEPLOYED graph needs scheduling, which is +#: tk-rebuild-full-text-indexes-on-the-deployed-witan--. Nothing here +#: performs it — this constant only decides which binary a future install or +#: image build fetches. +#: +#: CHECKS RUN, against the v0.10.0 binary (downloaded, digest-verified, run +#: from a scratch dir): +#: * Vocabulary — all 14 `_RETRYABLE`/`_NEEDS_REPAIR`/`_PRECONDITION_FAILED`/ +#: `_RECOVERY_REQUIRED` substrings against the 4-commit range. Every PROSE +#: marker (`stale view`, `omnigraph repair`, `refresh and retry`, +#: `reprepare from the current branch`, `write authority`, `ahead of +#: manifest`, ...) is untouched. `recovery_required`/`precondition_failure` +#: appear on both sides of `omnigraph-server/src/lib.rs` (+149/-271), a +#: refactor that preserves the HTTP field names rather than renaming them. +#: * Version/format — reads 0.10.0, internal-schema 6, unchanged. +#: * Suites against this binary: witan-core 584, witan-council 1009, +#: witan-code 580 — 2173 passed, NO failures and no artifact exclusions. +#: +#: ★ GETTING THAT NUMBER HONESTLY TAKES ONE STEP, and skipping it silently +#: tests the wrong binary. `testsupport/hermetic.py` PREPENDS the real +#: `~/.local/bin` to PATH (deliberately — see its docstring), so putting a +#: candidate binary earlier on PATH does NOT make the suite use it: the +#: machine's installed omnigraph still wins, and everything passes while +#: proving nothing about the new one. Run with `HOME` pointed at a scratch +#: dir holding the candidate at `$HOME/.local/bin/omnigraph`, with the real +#: `~/.local/bin` OFF PATH so only one omnigraph is reachable — otherwise +#: `test_pre_upgrade_candidates_exclude_the_current_binary` correctly +#: reports the second one and looks like a failure. +#: +#: Digests taken by downloading all three tarballs and hashing them locally, +#: cross-checked against each published `.sha256` — all three matched. On an +#: immutable tag the same-sitting caveat no longer bites. +#: +#: Going back to the moving tag means restoring the `edge` triple superseded +#: here, which was the ac620eea87 build: +#: linux-x86_64 6a0fba8842a2071c558abf2c1a399ce5e11d359dff78b6ae6ff3676617f95680 +#: linux-arm64 dd40fa4169a89af41cddbdeb8fe441b714438633297e153876b4889ec0af3a86 +#: macos-arm64 990fcab686922f885f959a0f6204f61d0770ef7af6f058bac9df14cc587a2248 _OMNIGRAPH_ASSET_SHA256: dict[str, str] = { "omnigraph-linux-x86_64.tar.gz": ( - "6a0fba8842a2071c558abf2c1a399ce5e11d359dff78b6ae6ff3676617f95680" + "05d3ce4ec0ab51a876befd89b643c3e7f2d5489be0398a38cef6fb3a0d257fc1" ), "omnigraph-linux-arm64.tar.gz": ( - "dd40fa4169a89af41cddbdeb8fe441b714438633297e153876b4889ec0af3a86" + "dd3ac09123a68882454db7e689da4c306c41677826237098df4e76b0f73d8d5e" ), "omnigraph-macos-arm64.tar.gz": ( - "990fcab686922f885f959a0f6204f61d0770ef7af6f058bac9df14cc587a2248" + "7c3b8fadbe590486a192c734d8c3d38cce0e4da1f02940e6ac306c1ada67f171" ), } _VERSION_RE = re.compile(r"\d+\.\d+\.\d+") From aa074ee2fd0af89d8f9c94840cbb21de538d56bd Mon Sep 17 00:00:00 2001 From: Christopher Patti Date: Mon, 31 Aug 2026 18:11:25 -0400 Subject: [PATCH 2/3] chore(witan-core): record the FTS-rebuild task slug in the pin history --- packages/witan-core/witan_core/omnigraph_install.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/witan-core/witan_core/omnigraph_install.py b/packages/witan-core/witan_core/omnigraph_install.py index d028c009..e6c580e6 100644 --- a/packages/witan-core/witan_core/omnigraph_install.py +++ b/packages/witan-core/witan_core/omnigraph_install.py @@ -424,7 +424,7 @@ #: Upstream calls it a controlled cutover, not a rolling upgrade: stop old #: readers/writers and keep a recoverable backup first. A local store is one #: command; the DEPLOYED graph needs scheduling, which is -#: tk-rebuild-full-text-indexes-on-the-deployed-witan--. Nothing here +#: tk-rebuild-full-text-indexes-on-the-deployed-witan--076eb6. Nothing here #: performs it — this constant only decides which binary a future install or #: image build fetches. #: From baa6a7571b944362c19f9bb03e9b397beef02550 Mon Sep 17 00:00:00 2001 From: Christopher Patti Date: Mon, 31 Aug 2026 18:23:19 -0400 Subject: [PATCH 3/3] fix(witan-core): classify omnigraph 0.10.0's full-text rebuild 409 as terminal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot review on #311, and it corrects the premise of the pin commit. omnigraph 0.10.0 does not ship raw Lance 11's silent under-return. Upstream #581 added a guard: an index whose analyzer generation cannot be proven compatible raises FullTextIndexRebuildRequired — HTTP 409 with a `full_text_index_rebuild_required` detail — and their doc comment is explicit that ordinary reads stay available and a partial indexed result is never returned. So search fails closed rather than degrading quietly. The history block said the opposite and is corrected. That makes the client the problem. `classify_status` treats a bare 409 as RETRYABLE on the status alone — deliberately, so a lost CAS race is legible — and nothing above it matched this message. Every refused search()/bm25() would therefore be retried the full budget and surface as a timeout-shaped error, hiding the remedy the server already printed. Adds FULL_TEXT_REBUILD_REQUIRED, classified ahead of the 409 catch-all on the structured detail key and on the server's own prose, terminal on the first attempt on both transports. Not NEEDS_REPAIR: the graph is not damaged and `omnigraph repair --force` would be the wrong hammer. Regression tests cover both transports, that the detail key alone suffices, and that an ordinary lost-race 409 is still RETRYABLE. All four fail without the classification. witan-core 588, witan-council 1009, witan-code 580 — 2177 passed against the v0.10.0 binary. --- packages/witan-core/tests/test_omnigraph.py | 33 +++++++++++++ .../witan-core/tests/test_omnigraph_http.py | 46 +++++++++++++++++++ packages/witan-core/witan_core/omnigraph.py | 30 ++++++++++++ .../witan-core/witan_core/omnigraph_http.py | 36 +++++++++++++++ .../witan_core/omnigraph_install.py | 38 ++++++++++----- 5 files changed, 172 insertions(+), 11 deletions(-) diff --git a/packages/witan-core/tests/test_omnigraph.py b/packages/witan-core/tests/test_omnigraph.py index e6190692..277ec637 100644 --- a/packages/witan-core/tests/test_omnigraph.py +++ b/packages/witan-core/tests/test_omnigraph.py @@ -1845,3 +1845,36 @@ def test_the_wait_is_capped_at_the_last_viable_moment(monkeypatch): elapsed = time.monotonic() - started assert elapsed < 2.0, "should give up near the viable boundary, not at 30s" assert elapsed > 0.2, "should have waited while admission was still viable" + + +# ── omnigraph 0.10.0's full-text rebuild guard (upstream #581) ────────────── +# +# Literal, for the reason `_WRITE_AUTHORITY_STDERR` is: taken from the `#[error]` +# attribute on `OmniError::FullTextIndexRebuildRequired` in +# crates/omnigraph/src/error.rs at v0.10.0. +_FTS_REBUILD_STDERR = ( + "omnigraph query failed:\n" + "full-text index 'memory_content' requires rebuild: analyzer generation " + "cannot be proven compatible; run omnigraph rebuild-full-text-indexes " + " --branch on the live branch (historical snapshots are " + "unchanged)" +) + + +def test_full_text_rebuild_required_is_terminal_on_the_cli_path(): + """Lance 11 changed the analyzer, so an index built by Lance 10 cannot serve + a 0.10.0 `search()`/`bm25()` query. Retrying never clears it — upstream says + so outright — so classifying it RETRYABLE would burn the whole attempt + budget and then report a timeout-shaped failure for a permanent condition + whose remedy is printed right there in the message. + """ + assert ( + og._classify_cli_error(_FTS_REBUILD_STDERR) == _http.FULL_TEXT_REBUILD_REQUIRED + ) + + +def test_full_text_rebuild_is_not_confused_with_a_repair(): + """NEEDS_REPAIR would run `omnigraph repair --force` on a graph that is not + damaged. Upstream is explicit that ordinary reads stay available and only + the full-text index is refused.""" + assert og._classify_cli_error(_FTS_REBUILD_STDERR) != _http.NEEDS_REPAIR diff --git a/packages/witan-core/tests/test_omnigraph_http.py b/packages/witan-core/tests/test_omnigraph_http.py index db0c9b54..a5624e03 100644 --- a/packages/witan-core/tests/test_omnigraph_http.py +++ b/packages/witan-core/tests/test_omnigraph_http.py @@ -227,6 +227,52 @@ def test_repair_still_wins_over_a_conflict_status(): assert ogh.classify_status(409, message) == ogh.NEEDS_REPAIR +def test_full_text_rebuild_required_is_terminal_not_a_retryable_conflict(): + """omnigraph 0.10.0 (upstream #581) answers a stale full-text index with a + 409 carrying `full_text_index_rebuild_required`. The bare-409 rule below it + would retry that to exhaustion and bury the remedy, so this must be caught + on the structured key AND on the server's prose.""" + body = json.dumps( + { + "error": ( + "full-text index 'memory_content' requires rebuild: analyzer " + "generation cannot be proven compatible; run omnigraph " + "rebuild-full-text-indexes --branch on the live " + "branch (historical snapshots are unchanged)" + ), + "code": "conflict", + "details": { + "full_text_index_rebuild_required": { + "index": "memory_content", + "reason": "analyzer generation cannot be proven compatible", + } + }, + } + ) + assert ogh.classify_status(409, body) == ogh.FULL_TEXT_REBUILD_REQUIRED + assert ogh.classify_status(409, body) != ogh.RETRYABLE + + # The detail key alone is enough — a relay that drops the prose still + # classifies correctly. + assert ( + ogh.classify_status( + 409, '{"details": {"full_text_index_rebuild_required": {}}}' + ) + == ogh.FULL_TEXT_REBUILD_REQUIRED + ) + + +def test_a_plain_conflict_is_still_retryable(): + """The guard above must not swallow the ordinary lost-race 409 that + `task_claim` depends on.""" + message = ( + "write authority 'graph_head:main' changed during preparation " + "(expected 01M08E24Y, current 01M08E27K) - reprepare from the " + "current branch state" + ) + assert ogh.classify_status(409, message) == ogh.RETRYABLE + + def test_error_message_uses_the_servers_own_wording(): body = json.dumps({"error": "policy denied action 'change'", "code": "forbidden"}) assert ogh.error_message(403, body) == ( diff --git a/packages/witan-core/witan_core/omnigraph.py b/packages/witan-core/witan_core/omnigraph.py index 9a043950..70cb87df 100644 --- a/packages/witan-core/witan_core/omnigraph.py +++ b/packages/witan-core/witan_core/omnigraph.py @@ -146,6 +146,21 @@ # node:ClaimRow); reopen the graph read-write before retrying". Matching the # stem they share rather than either sentence. _RECOVERY_REQUIRED = ("recovery required", "recovery_required") + +# Checked BEFORE _RETRYABLE for the same reason the HTTP side checks it before +# its 409 catch-all: this is terminal, and retrying only delays the one message +# that says what to do. omnigraph 0.10.0 prints, verbatim: +# +# full-text index '' requires rebuild: ; run omnigraph +# rebuild-full-text-indexes --branch on the live branch +# (historical snapshots are unchanged) +# +# The command name is the marker rather than "requires rebuild" alone, which is +# generic enough to collide with a future unrelated message. +_FULL_TEXT_REBUILD_REQUIRED = ( + "rebuild-full-text-indexes", + "full_text_index_rebuild_required", +) _MAX_ATTEMPTS = 8 # omnigraph uses strict single-version storage: a release that bumps the @@ -524,6 +539,8 @@ def _classify_cli_error(stderr: str) -> str: return _http.PRECONDITION_FAILED if any(m in lowered for m in _RECOVERY_REQUIRED): return _http.RECOVERY_REQUIRED + if any(m in lowered for m in _FULL_TEXT_REBUILD_REQUIRED): + return _http.FULL_TEXT_REBUILD_REQUIRED if any(m in lowered for m in _NEEDS_REPAIR): return _http.NEEDS_REPAIR if any(m in lowered for m in _RETRYABLE): @@ -2086,6 +2103,19 @@ def _retry_loop( f"you read it. NOTHING WAS WRITTEN, and this write must " f"not be retried as-is; re-read and decide:\n{err.strip()}" ) + if kind == _http.FULL_TEXT_REBUILD_REQUIRED: + # TERMINAL, on the first attempt. The index cannot serve + # this engine's analyzer and no amount of retrying changes + # that; upstream says so outright. Ordinary reads are + # unaffected — only full-text queries land here — so this is + # not a broken graph, and the remedy is in the server's own + # message, which is why it is passed through whole. + raise RuntimeError( + f"omnigraph {label} was refused because a full-text " + f"index needs an explicit rebuild for this engine " + f"version. Retrying will not clear it, and ordinary " + f"(non-search) reads are unaffected:\n{err.strip()}" + ) if surface_conflict and kind == _http.RETRYABLE: # A compare-and-swap caller wants to lose the race, not # re-apply its write over the winner. Surface immediately. diff --git a/packages/witan-core/witan_core/omnigraph_http.py b/packages/witan-core/witan_core/omnigraph_http.py index 308c489a..e2a970c6 100644 --- a/packages/witan-core/witan_core/omnigraph_http.py +++ b/packages/witan-core/witan_core/omnigraph_http.py @@ -79,6 +79,30 @@ #: classification is what makes that true on our side too. PRECONDITION_FAILED = "precondition_failed" +#: A full-text index whose analyzer generation cannot be proven compatible with +#: the running engine — omnigraph 0.10.0's ``FullTextIndexRebuildRequired``, +#: answered as HTTP 409 with a ``full_text_index_rebuild_required`` detail +#: object (upstream #581). +#: +#: ★ TERMINAL, AND IT HAS TO BE NAMED BECAUSE 409 ALREADY MEANS "RETRY ME". +#: :func:`classify_status` treats a bare 409 as :data:`RETRYABLE` on the status +#: alone — deliberately, see the comment there — so without this check first, +#: every BM25/``search()`` request against a pre-upgrade index would be retried +#: the full budget and then surface as a generic retry-exhausted error, hiding +#: the one sentence that says what to do. Upstream is explicit that it "is not +#: cleared by retrying". +#: +#: ★ IT IS NOT A BROKEN GRAPH, and specifically not :data:`NEEDS_REPAIR`. +#: Upstream's own doc comment on the variant: "Ordinary reads remain available; +#: do not return a partial indexed result." Only full-text queries are refused, +#: and the fix is one explicit command per branch: +#: +#: omnigraph rebuild-full-text-indexes --branch +#: +#: Running ``omnigraph repair`` here would be a much heavier hammer aimed at +#: the wrong thing. +FULL_TEXT_REBUILD_REQUIRED = "full_text_rebuild_required" + #: The branch-wide write barrier: "recovery required for operation …: pending #: Load recovery operation blocks writes on branch 'main'". #: @@ -236,6 +260,18 @@ def classify_status(status: int, message: str) -> str: for marker in ("stale view", "manifest table version", "refresh and retry") ): return RETRYABLE + # BEFORE the 409 catch-all below, which would otherwise retry this to + # exhaustion — see FULL_TEXT_REBUILD_REQUIRED. The structured detail key is + # the primary signal because it is a wire contract; the prose is checked too + # so the CLI path and any non-JSON relay reach the same answer. + if any( + marker in lowered + for marker in ( + "full_text_index_rebuild_required", + "rebuild-full-text-indexes", + ) + ): + return FULL_TEXT_REBUILD_REQUIRED if status == 409: # ★ A CONFLICT IS THE ONE STATUS THAT MEANS "RETRY ME" MOST LITERALLY, # and it was falling through to FATAL because this function only ever diff --git a/packages/witan-core/witan_core/omnigraph_install.py b/packages/witan-core/witan_core/omnigraph_install.py index e6c580e6..335a6c20 100644 --- a/packages/witan-core/witan_core/omnigraph_install.py +++ b/packages/witan-core/witan_core/omnigraph_install.py @@ -410,17 +410,26 @@ #: Upstream's own cross-version v0.9->v0.10 suite plus release notes. #: #586 `test(bench): remove synthetic worker timing race` — tests only. #: -#: ★ LANCE 11 CHANGES ENGLISH STEMMING, AND THE STORAGE-FORMAT GATE WILL NOT -#: TELL YOU. `_OMNIGRAPH_INTERNAL_SCHEMA` stays 6 and `omnigraph version` on -#: this binary agrees, so `bin/check_omnigraph_format.py` is green — correctly, -#: because the on-disk format did not move. What DID move is the full-text -#: analyzer: an index built by Lance 10 read with the Lance 11 analyzer SILENTLY -#: UNDER-RETURNS (upstream's own regression cites `organism` and `university`). +#: ★ LANCE 11 CHANGES THE ANALYZER, AND FULL-TEXT SEARCH FAILS CLOSED UNTIL +#: EACH BRANCH IS REBUILT. `_OMNIGRAPH_INTERNAL_SCHEMA` stays 6 and `omnigraph +#: version` on this binary agrees, so `bin/check_omnigraph_format.py` is green +#: — correctly, because the on-disk format did not move. What moved is the +#: full-text analyzer. +#: +#: The silent-under-return that motivated upstream #581 (their regression cites +#: `organism` and `university`) is what RAW Lance does on a generation +#: mismatch. 0.10.0 does NOT ship that behaviour: it added a guard. A selected +#: index whose analyzer generation cannot be proven compatible raises +#: `OmniError::FullTextIndexRebuildRequired` — HTTP 409 with a +#: `full_text_index_rebuild_required` detail, and upstream's own doc comment +#: says "Ordinary reads remain available; do not return a partial indexed +#: result". So search is UNAVAILABLE, not quietly worse, and non-search reads +#: are untouched. +#: #: Every `search()`/`bm25()` query in `read.gq` sits on such an index — memory #: search and the Task/WorkflowProject BM25 search both — so on any graph -#: written before this binary, search quietly gets worse rather than failing. -#: The remedy is explicit and per branch: -#: omnigraph rebuild-full-text-indexes --branch +#: written before this binary, those queries are refused until: +#: omnigraph rebuild-full-text-indexes --branch #: Upstream calls it a controlled cutover, not a rolling upgrade: stop old #: readers/writers and keep a recoverable backup first. A local store is one #: command; the DEPLOYED graph needs scheduling, which is @@ -428,6 +437,13 @@ #: performs it — this constant only decides which binary a future install or #: image build fetches. #: +#: ★ AND THE CLIENT HAD TO LEARN THAT 409 FIRST, which is why this commit is +#: not digests alone. `classify_status` treats a bare 409 as RETRYABLE on the +#: status, so an un-taught client would retry every refused search the full +#: budget and then report a timeout-shaped failure, burying the remedy the +#: server already printed. `_http.FULL_TEXT_REBUILD_REQUIRED` classifies it +#: terminal on both transports. +#: #: CHECKS RUN, against the v0.10.0 binary (downloaded, digest-verified, run #: from a scratch dir): #: * Vocabulary — all 14 `_RETRYABLE`/`_NEEDS_REPAIR`/`_PRECONDITION_FAILED`/ @@ -438,8 +454,8 @@ #: appear on both sides of `omnigraph-server/src/lib.rs` (+149/-271), a #: refactor that preserves the HTTP field names rather than renaming them. #: * Version/format — reads 0.10.0, internal-schema 6, unchanged. -#: * Suites against this binary: witan-core 584, witan-council 1009, -#: witan-code 580 — 2173 passed, NO failures and no artifact exclusions. +#: * Suites against this binary: witan-core 588, witan-council 1009, +#: witan-code 580 — 2177 passed, NO failures and no artifact exclusions. #: #: ★ GETTING THAT NUMBER HONESTLY TAKES ONE STEP, and skipping it silently #: tests the wrong binary. `testsupport/hermetic.py` PREPENDS the real