Rust MongoDB server: fix-one batch snapshot of rm-batch (109f5401, 1 commits)

.claude/skills/batch-worktree/SKILL.md

@@ -274,6 +274,8 @@ Why batches: nine
   - `tests/test_no_tracked_build_output.py` fails on a tracked cargo target
     directory or any tracked file over 8 MB. Run it when in doubt; do not
     raise its limit to get something in.
-  - The one binary that IS committed on purpose is the PGO profile,
-    `crates/pgo/_secantus_server.profdata.tar.gz` (see Tooling). Adding
-    another needs Joe's say-so.
+  - The binaries that ARE committed on purpose: the PGO profile,
+    `crates/pgo/_secantus_server.profdata.tar.gz` (see Tooling), and mongo's
+    trimmed ICU data, `vendor/icu4c-57.1/source/mongo_sources/icudt57l.dat`
+    and `icudt57b.dat` (Joe, 2026-11-09: mongod's own data, built into
+    `crates/secantus-icu`). Adding another needs Joe's say-so.

.claude/skills/build-and-test/SKILL.md

@@ -58,6 +58,7 @@ binary built from neither tree; the provenance check will refuse it.
 
 ## WiredTiger
 
+- **ICU 57.1 is vendored** at `vendor/icu4c-57.1` (committed, not a submodule): mongo r8.2.11's own `src/third_party/icu4c-57.1`, sources and trimmed data, as Joe decided 2026-11-09. `crates/secantus-icu` compiles it with `cc` from the source lists in its `source/BUILD.bazel`, so every crate that links `secantus-core` (both servers, the PyO3 bindings) builds ~800 C/C++ files once per target dir, about a minute at 3 jobs. Never edit the copy; refresh it whole from mongo's tree. `scripts/crates_package_check.py` copies it into the packaged crate. The build passes `inherit_rustflags(false)`, as the lz4 build does, so the PGO flags never reach Apple clang.
 - WiredTiger is **vendored** as a git submodule at `vendor/wiredtiger` (mongodb-7.0.33). The CMake build is driven by `CMakeLists.txt` (scikit-build-core + ExternalProject) and produces self-contained binary wheels via `cibuildwheel` for **cp310–cp313** on macOS arm64, **manylinux_2_28** + musllinux_1_2 x86_64/aarch64, and Windows AMD64 (read off the published 0.6.0b17 wheel list on 2026-09-27 — this line previously said cp312+cp313 / manylinux2014, which had drifted). macOS x86_64 is intentionally absent (runner-pool scarcity, Apple Silicon is the active target). `pip install SecantusDB` ships pre-built on supported platforms; users never need `cmake`/`ninja`/`swig`. The `cmake/patch_wt_*.py` scripts apply small idempotent patches to the vendored WT tree at `PATCH_COMMAND` time (off64_t→off_t for musl, Python module SUFFIX/dynamic_lookup, etc.); fix WT-side incompat by extending one of these patchers, not by editing the submodule directly. **Two patchers change BEHAVIOUR rather than the build**: `patch_wt_dsync_group.py` (2026-10-07, Joe's call) makes concurrent `method=dsync` commits share log writes (only the PostgreSQL server on macOS uses dsync), and `patch_wt_fsync_group.py` (2026-10-09) credits one `fsync` under `method=fsync` with every commit already written, which every server that syncs per commit off macOS runs. A process kill cannot test the second one, because the page cache survives it: its test is a hard reboot under load (`tasks/backlog-rust-mongodb.md`, the Linux entry). Three builds apply the patches and must agree — the wheel's CMake (`apply_wt_patches.cmake`), the crate's bundled copy (`scripts/wt_sys_refresh.py`, then commit `wiredtiger.sha256`) and `./inv rust-wt-build` (which reads the crate's list) — so a new patcher goes in the first two lists.
 
 ## Profile-guided optimisation

.github/workflows/release-binaries.yml

@@ -452,6 +452,9 @@ jobs:
           [ "${{ matrix.os }}" = "windows" ] && exe=".exe"
           cp "$BIN" "$stem/secantusd-rs$exe"
           cp LICENSE README.md "$stem/"
+          # ICU 57.1 is linked in (crates/secantus-icu); its licence asks
+          # for its notice to travel with the binary.
+          cp vendor/icu4c-57.1/LICENSE "$stem/LICENSE-ICU"
 
           # `shasum` ships with macOS/Linux runners but not Git Bash on Windows,
           # which has `sha256sum` instead. Pick whichever exists rather than

.github/workflows/release-pg-binaries.yml

@@ -247,6 +247,9 @@ jobs:
           mkdir -p "$stem"
           cp "$BIN" "$stem/secantusd-pg"
           cp LICENSE README.md "$stem/"
+          # ICU 57.1 is linked in (crates/secantus-icu); its licence asks
+          # for its notice to travel with the binary.
+          cp vendor/icu4c-57.1/LICENSE "$stem/LICENSE-ICU"
           if command -v shasum >/dev/null 2>&1; then
             sha() { shasum -a 256 "$1"; }
           else

changelog.d/rm-095-icu57-collation.md

@@ -0,0 +1,59 @@
+### The Rust MongoDB server collates with mongod's own ICU
+
+Every collated comparison in mongod is ICU 57.1's answer, and now so is the
+Rust MongoDB server's. It links the same ICU, built from mongo's own copy of
+the sources and its trimmed locale data, and makes the calls mongod makes. Before,
+the server used ICU4X, a Rust reimplementation with newer data. ICU4X
+agreed with mongod on order but could not do three things. It could not tell
+two canonically equivalent spellings apart under `normalization: false`, the
+default. Its data lacked five collation types mongod accepts: `search`,
+`searchjl`, `big5han`, `gb2312han` and `phonetic`. And its sort keys had
+different bytes, so a duplicate-key error quoted a different key.
+
+What a locale defaults to now comes from ICU too, as it does on mongod: an
+index on `{locale: "da"}` sorts capitals first and says so in
+`listIndexes`, a `fr_CA` query at strength 1 is refused because that locale
+compares accents backwards, and which locale names are valid, with the
+"Did you mean" each refusal suggests, is ICU's reading of the name. A store
+written before this change has its collated indexes and collated `_id`
+claims re-keyed the first time it opens.
+
+#### Fixed
+
+- `crates/secantus-icu` (new): builds mongo r8.2.11's
+  `src/third_party/icu4c-57.1`, vendored at `vendor/icu4c-57.1`, from the
+  source lists, defines and warning switches of its `BUILD.bazel`, and hands
+  ICU the embedded `icudt57l.dat` (`icudt57b.dat` on big-endian) through
+  `udata_setCommonData`, as mongod's `icu_init.cpp` does. A C++ shim makes
+  mongod's `icu::Collator` calls (`createInstance`, `get/setAttribute`,
+  `setMaxVariable`, `compareUTF8`, `getCollationKey`).
+- `secantus-core::collation`: compares, sorts and keys through it. The spec
+  is resolved as `updateCollationSpecFromICUCollator` does it: a given
+  option is set, an absent one is read from the locale. `create` fills the
+  IDL defaults first, as mongod does. The spec checks run on the resolved
+  options. Locale validation is ICU's (`ULOC_VALID_LOCALE` against the name
+  given), replacing the hand-built locale table and its three-letter-code
+  reconstruction. Strings that are not UTF-8 compare through
+  `compareUTF8`, with each invalid sequence weighed as U+FFFD.
+- Sort keys are mongod's bytes (`"W\x01\x05"` for `"x"` at `en` strength
+  2). `collation::KEY_FORMAT` is 2: collated indexes built earlier are
+  rebuilt on open. The collated `_id` claim marker now records its format,
+  and claims under an older one are dropped and taken again, so a second
+  spelling of a stored `_id` is still refused.
+- Measured against mongod 8.2.11: 2,202 locale, type, default and refusal
+  cases all equal, and `tools/probes/collation.py` down from 4 divergent
+  shapes to 2 (the unbuilt hashed and text indexes), now with sort-key
+  bytes compared, not scrubbed. A duplicate document `_id` (RM-092) now
+  quotes mongod's exact key bytes, and its gap test asserts them.
+- Fixes RM-095 (tests/backlog/test_rm_085_112.py::test_five_collation_types_are_accepted)
+  and RM-091 (tests/backlog/test_rm_085_112.py::test_canonically_equivalent_strings_differ_without_normalization).
+  Pinned by `tests/mongod_live/test_collation_icu.py`, the `secantus-icu` and
+  `collation` unit tests, and
+  `secantus-storage`'s `open_rekeys_collated_id_claims_of_an_older_key_format`.
+- The PostgreSQL server links `secantus-core` and so builds ICU 57.1 too.
+  Its own collations still go through `secantus-pgplan`'s ICU4X.
+- `tools/probes/_servers.py`: a replica-set probe now starts the embedded
+  Rust server with the same replica set. It had been compared as a
+  standalone against a replica-set mongod, and `time_series.py` showed
+  22 false divergences, all `commitQuorum`.
+- The release archives ship ICU's licence as `LICENSE-ICU`.

crates/Cargo.toml

@@ -31,6 +31,7 @@
 resolver = "2"
 members = [
     "secantus-core",
+    "secantus-icu",
     "secantus-core-py",
     "secantus-wire",
     "secantus-commands",

crates/secantus-core/Cargo.toml

@@ -37,17 +37,10 @@ snap = "1"
 # pattern matching at each length; `regex` only offers leftmost-first. Already
 # in the lock tree as a dependency of `regex`.
 regex-automata = "0.4"
-# Collation. ICU4X is the Rust implementation of the algorithm and data
-# mongod gets from ICU; see `src/collation.rs` for how closely the two were
-# measured to agree. `unstable` is what exposes sort keys, which an index
-# entry and a group key need. The versions are pinned exactly because a sort
-# key is written to disk (`collation::KEY_FORMAT`): a data update that changes
-# its bytes has to be a deliberate bump, not something `cargo update` does.
-icu_collator = { version = "=2.3.1", features = ["unstable"] }
-icu_collator_data = "=2.3.0"
-icu_normalizer = "2"
-icu_locale_core = "2"
-icu_provider = { version = "2", features = ["sync"] }
+# Collation: the ICU 57.1 mongod links, built from mongo's own copy
+# (`vendor/icu4c-57.1`). A sort key is written to disk
+# (`collation::KEY_FORMAT`), and this is the collator that makes its bytes.
+secantus-icu = { version = "=0.5.3-beta.196", path = "../secantus-icu" }
 # Backtracking regex with lookaround + backreference support (Python `re`-style),
 # used only when the linear `regex` engine above can't compile a pattern — e.g.
 # the negative-lookahead `^(?!system\.)` pymongo emits for listCollections name