Rust MongoDB server: fix-one batch snapshot of rm-batch (d35154a7, 5 commits)

.claude/skills/rust-mongodb-explain/SKILL.md

@@ -3,4 +3,4 @@ name: rust-mongodb-explain
 description: 'How the Rust MongoDB server reproduces mongod explain, the classic plan cache and document return order (crates/secantus-commands/src/explain/). Fires when touching explain, the planner, the plan cache, the storage router, or result order on the Rust MongoDB server.'
 ---
 
-**The Rust MongoDB server reproduces mongod's `explain` (2026-10-12)** -- `indexBounds`, the `IDHACK` / `EXPRESS_IXSCAN` / `COUNT_SCAN` / `DISTINCT_SCAN` / `PROJECTION_COVERED` executors and every command's stages, because they turned out to follow fixed rules whenever only one index can serve the query: its `explain` (`crates/secantus-commands/src/explain/`) derives the plan from mongod's planning rules — bounds, residual filters, sort and covering, and the `count` / `distinct` / write stages — and derives `executionStats` by walking that plan over the documents. Since 2026-10-27 it also keeps mongod 8.2's classic plan cache (`explain/plan_cache.rs`, `explain/cache_key.rs`, RM-154): the query-shape key and its hashes byte for byte, inactive and active entries, replanning, the 5,000-entry eviction, what clears it, and `$planCacheStats`; the trial ranks plans with mongod's tie-breaking bonuses. A pipeline front SBE runs (`plan::Query::sbe`, since 2026-11-08, RM-340) is planned without index intersections and cached under its `s` key by the keys and documents its winner read (`worksType: "reads"`), replanned when its reads reach ten times that (`trial::within_reads`). Not yet: a rooted `$or`'s branches in such a front (RM-429). Past 2,000 documents a trial no longer reads the collection whole: it runs over what its candidates' scans read from the store in the works it takes (`order::Lazy::reached`, `stream::scan_reach`), read again for eight times as many works when it ran out of them (`plan::over_reach`; RM-342). It reports the plan mongod runs, which is not always how the Rust storage router reads the documents (`tasks/backlog-rust-mongodb.md` 7.075). Sweep: `tools/probes/explain_commands.py`, 786 replies. **Since 2026-10-13 the same plan sets the ORDER the Rust server returns documents in** (`explain/order.rs`, `tasks/backlog-rust-mongodb.md` 7.080): the router says only which documents match, and the plan says in what order mongod reads them, which is also which document a `limit: 1` delete or an update without `multi` changes. A sort is still the store's own; the plan decides how documents that compare equal are left. For a sort the predicate's index does not return, the planner counts mongod's trial of the two candidate plans (`plan::counted`, else the trial) on a collection of any size. Sweep: `tools/probes/find_order.py`, 6,531 replies. When a change touches the router or the planner, run it: a router change can no longer move the order, and a planner change now can.
+**The Rust MongoDB server reproduces mongod's `explain` (2026-10-12)** -- `indexBounds`, the `IDHACK` / `EXPRESS_IXSCAN` / `COUNT_SCAN` / `DISTINCT_SCAN` / `PROJECTION_COVERED` executors and every command's stages, because they turned out to follow fixed rules whenever only one index can serve the query: its `explain` (`crates/secantus-commands/src/explain/`) derives the plan from mongod's planning rules — bounds, residual filters, sort and covering, and the `count` / `distinct` / write stages — and derives `executionStats` by walking that plan over the documents. Since 2026-10-27 it also keeps mongod 8.2's classic plan cache (`explain/plan_cache.rs`, `explain/cache_key.rs`, RM-154): the query-shape key and its hashes byte for byte, inactive and active entries, replanning, the 5,000-entry eviction, what clears it, and `$planCacheStats`; the trial ranks plans with mongod's tie-breaking bonuses. A pipeline front SBE runs (`plan::Query::sbe`, since 2026-11-08, RM-340) is planned without index intersections and cached under its `s` key by the keys and documents its winner read (`worksType: "reads"`), replanned when its reads reach ten times that (`trial::within_reads`). A rooted `$or`'s branches in such a front are planned in the classic engine and cached with `worksType: "works"` under the subplanner's `o` key (RM-429). Past 2,000 documents a trial no longer reads the collection whole: it runs over what its candidates' scans read from the store in the works it takes (`order::Lazy::reached`, `stream::scan_reach`), read again for eight times as many works when it ran out of them (`plan::over_reach`; RM-342). It reports the plan mongod runs, which is not always how the Rust storage router reads the documents (`tasks/backlog-rust-mongodb.md` 7.075). Sweep: `tools/probes/explain_commands.py`, 786 replies. **Since 2026-10-13 the same plan sets the ORDER the Rust server returns documents in** (`explain/order.rs`, `tasks/backlog-rust-mongodb.md` 7.080): the router says only which documents match, and the plan says in what order mongod reads them, which is also which document a `limit: 1` delete or an update without `multi` changes. A sort is still the store's own; the plan decides how documents that compare equal are left. For a sort the predicate's index does not return, the planner counts mongod's trial of the two candidate plans (`plan::counted`, else the trial) on a collection of any size. Sweep: `tools/probes/find_order.py`, 6,531 replies. When a change touches the router or the planner, run it: a router change can no longer move the order, and a planner change now can.

changelog.d/rm-429-or-branch-sbe.md

@@ -0,0 +1,14 @@
+### Rust MongoDB server: a rooted `$or`'s branches are cached in front of SBE stages
+
+A rooted `$or` in front of stages mongod runs in SBE (a `$group`, say) now
+caches each branch the way mongod 8.2.11 does: a classic entry
+(`worksType: "works"`) under a key whose shape ends in `o`, the flag mongod's
+`encodeClassic` gives a subplanner's branch of an SBE-compatible query, so it
+never shares an entry with a classic query of the same clause.
+
+#### Fixed
+
+- `$planCacheStats` lists the branch entries of such a pipeline, with
+  mongod's `planCacheShapeHash` and `planCacheKey`.
+
+Fixes RM-429 (tests/backlog/test_plan_cache_or_branches.py::test_an_or_branch_before_sbe_stages_is_cached)

changelog.d/rm-433-compound-scan-keys.md

@@ -0,0 +1,18 @@
+### Rust MongoDB server: a compound index scan over several intervals examines mongod's keys
+
+A scan of `a_1_c_1` for `{a: {$in: [3, 60]}, c: {$in: [1, 5]}}` now
+examines 16 keys with 5 seeks, as mongod 8.2.11's does, in `explain` and in
+the plan-cache trial whose works score the candidates.
+
+#### Fixed
+
+- Past the last interval of a later field, the scan ends when every earlier
+  field is on the point that is its last interval, as mongod's
+  `IndexBoundsChecker` does, instead of examining one more key.
+- A seek to a field's next interval lands at the start of the later fields'
+  first intervals, so the keys before them are not examined. A losing
+  `$nin` candidate over three intervals of one field now seeks and returns
+  as mongod's does in the trial, and scores the same.
+
+Fixes RM-433 (tests/backlog/test_rm_found_2026_11_08_rm342.py::test_a_scan_of_two_in_lists_examines_mongods_keys)
+Fixes RM-438 (tests/backlog/test_rm_found_2026_11_08_rm341.py::test_a_nin_candidate_reads_mongods_keys)

changelog.d/rm-434-explain-work-accounting.md

@@ -0,0 +1,16 @@
+### Rust MongoDB server: `explain`'s winning plan reports its work accounting
+
+`executionStats` now gives every stage of the winning plan the counters
+mongod 8.2.11 gives: `works`, `advanced`, `needTime`, `needYield`,
+`saveState`, `restoreState` and `isEOF`, and an index scan's `seeks`,
+`dupsTested` and `dupsDropped`. Before, a stage had only its `nReturned` and
+examined counts.
+
+#### Fixed
+
+- A `find`, `distinct` or aggregate plan's `executionStages` carry each
+  stage's work accounting, counted by working the plan to its end as mongod
+  works the winner. Under an aggregate's `$cursor`, `saveState` and
+  `restoreState` count the batches the cursor loaded.
+- Fixes RM-434 (tests/backlog/test_rm_found_2026_11_08_rm342.py::test_explain_gives_each_stage_its_work_accounting, tests/backlog/test_rm_found_2026_11_08_rm310.py::test_a_distinct_scan_run_reports_its_work_counters)
+- Filed RM-692: a `$cursor` that the pipeline stops reading still reports a plan run to its end.

changelog.d/rm-435-sbe-group-sort-size.md

@@ -0,0 +1,18 @@
+### Rust MongoDB server: a `$sort` after a `$group` run in SBE counts mongod's bytes
+
+A `$sort` behind a `$group` that mongod 8.2 runs in SBE now reports the
+`totalDataSizeSortedBytesEstimate` mongod does: SBE hands each group to the
+classic `$sort` as a document built field by field, each sub-document its
+own built storage, which costs more than the document the classic engine
+builds.
+
+#### Fixed
+
+- `[{$group: {_id: "$a"}}, {$sort: {_id: 1}}]` over ten documents counted
+  351 bytes where mongod 8.2.11 counts 735.
+  Fixes RM-435 (tests/backlog/test_rm_found_2026_11_08_rm071.py::test_a_sort_after_a_pushed_down_group_sizes_what_sbe_hands_it)
+
+#### Found
+
+- RM-693: a `$group` with `$mod` in its `_id`, or an expression object in an
+  accumulator, is not pushed into SBE.

changelog.d/rm-436-plan-cache-size-or-elemmatch.md

@@ -0,0 +1,15 @@
+### Rust MongoDB server: `$planCacheStats` sizes `$or` and `$elemMatch` entries
+
+A plan cache entry whose winner reads an `OR` of index scans beside other
+clauses, or whose query holds an `$elemMatch` its index does not read, now
+reports mongod 8.2.11's `estimatedSizeBytes` on macOS arm64 instead of
+leaving the field out.
+
+#### Fixed
+
+- The entry's `PlanCacheIndexTree` is built for an `OR` over single-predicate
+  branches (each tagged with the index it reads, `tagForSort` taking the
+  lowest-numbered) and for an `ELEM_MATCH_VALUE` node, and an `OR` stage's
+  `OrStats` is sized.
+
+Fixes RM-436 (tests/backlog/test_rm_found_2026_11_08_rm341.py::test_entries_this_does_not_size_report_their_size)

crates/secantus-commands/src/explain/cache_key.rs

@@ -763,6 +763,9 @@ pub(crate) struct Parts<'a> {
     pub distinct: Option<&'a str>,
     /// Planned by the classic planner and run in SBE.
     pub sbe: bool,
+    /// A rooted `$or`'s branch, which the classic subplanner plans and
+    /// caches (`CanonicalQuery::forSubPlanner`).
+    pub subplanner: bool,
 }
 
 /// `encodeClassic`: the query shape.
@@ -780,7 +783,15 @@ pub(crate) fn shape(e: &Expr, p: &Parts) -> String {
         out.push('0');
     }
     out.push('f');
-    out.push(if p.sbe { 's' } else { 'c' });
+    // `c` for a plan the classic engine runs, `s` for one planned by the
+    // classic planner and run in SBE, and `o` for a subplanner's branch of
+    // a query SBE runs: its entry stores `works`, and the flag keeps a
+    // classic query from sharing it (mongod 8.2.11, RM-429).
+    out.push(match (p.sbe, p.subplanner) {
+        (false, _) => 'c',
+        (true, false) => 's',
+        (true, true) => 'o',
+    });
     out
 }
 
@@ -1037,6 +1048,27 @@ mod tests {
         assert_eq!(hex(compute_hash("res/im/||||fc")), "4F486929");
     }
 
+    /// A subplanner's branch of a query SBE runs ends its shape in `o`
+    /// (mongod 8.2.11, 2026-11-14: the branch `{a: {$lt: 3}, b: {$lt: 50}}`
+    /// of a rooted `$or` in front of a `$group`, over `a_1` and `b_1`, RM-429).
+    #[test]
+    fn a_subplanner_branch_of_an_sbe_query_is_keyed_o() {
+        let e = parse(&doc! { "a": {"$lt": 3}, "b": {"$lt": 50} }).unwrap();
+        let projection = doc! { "b": 1, "_id": 0 };
+        let parts = Parts {
+            sort: None,
+            projection: Some(&projection),
+            collation: None,
+            distinct: None,
+            sbe: true,
+            subplanner: true,
+        };
+        let s = shape(&e, &parts);
+        assert_eq!(s, "an[lta,ltb]||b||fo");
+        assert_eq!(hex(compute_hash(&s)), "EDB869B0");
+        assert_eq!(hex(key_hash(&format!("{s}<1><1>"))), "CA4248A6");
+    }
+
     /// The read's query settings move the key, not the shape hash (mongod
     /// 8.2.11, 2026-11-10: `find({x: 1})`, no index, RM-418).
     #[test]
@@ -1095,6 +1127,7 @@ mod tests {
             collation: None,
             distinct: None,
             sbe: false,
+            subplanner: false,
         };
         let shape_of = |q: Document| shape(&parse(&q).unwrap(), &none);
         assert_eq!(

crates/secantus-commands/src/explain/change_stream.rs

@@ -748,6 +748,7 @@ pub(super) fn reply(
             collation: None,
             distinct: None,
             sbe: false,
+            subplanner: false,
         },
         &cache_key::Indexability {
             catalog: &[],
@@ -874,6 +875,7 @@ mod tests {
                 collation: None,
                 distinct: None,
                 sbe: false,
+                subplanner: false,
             },
             &cache_key::Indexability {
                 catalog: &[],