Rust MongoDB server: fix-one batch snapshot of rm-batch (1f93877d, 4 commits)

changelog.d/rm-087-conflict-cause-and-null-stmtids.md

@@ -0,0 +1,16 @@
+### Rust MongoDB server: an insert's write conflict names the `_id` index, and a retryable insert does not count a null
+
+A transaction's insert that conflicts on an `_id` another open transaction
+inserted now answers 112 with mongod 8.2.11's cause, naming the index and the
+table it conflicted in (`WiredTigerIdIndex::_insert: index: _id_; uri:
+table:<ident>`), where the cause was empty. A retryable insert drops a null
+from its `documents` before counting them against `stmtIds`, as mongod does,
+instead of refusing the write (16).
+
+#### Fixed
+
+- Fixes RM-087 (tests/backlog/test_rm_085_112.py::test_an_insert_conflict_names_the_index_table)
+- Fixes RM-087 (tests/backlog/test_rm_085_112.py::test_a_retryable_insert_does_not_count_a_null)
+
+The storage change (`secantus-storage`'s write-conflict cause) is linked by the
+PostgreSQL server too; it reads nothing new from it.

changelog.d/rm-456-time-series-inclusion-project.md

@@ -0,0 +1,19 @@
+### Rust MongoDB server: a time-series `aggregate` behind an inclusion `$project` reads a bucket at a time
+
+A time-series `aggregate` of `$match` and an inclusion `$project` now unpacks
+its buckets one at a time as the cursor asks, with no cursor-stage buffer, as
+mongod 8.2.11's slot-based engine does; an inclusion `$project` alone reads
+through the 32-bucket buffer like the other streamed stages. Both used to read
+the whole collection at the command.
+
+#### Fixed
+
+- A time-series cursor behind an inclusion `$project` returned 3,999 more
+  measurements after a delete where mongod returns 71 (with a `$match`) or
+  2,303 (without one).
+
+Files RM-702 (an insert racing the TTL monitor fails with 112).
+
+Fixes RM-456 (tests/backlog/test_rm_found_2026_11_09_rm348.py::test_a_time_series_projection_reads_a_bucket_at_a_time)
+
+Filed RM-702: an insert into a time-series bucket the TTL monitor removes fails with 112.

changelog.d/rm-457-ts-meta-index-scan.md

@@ -0,0 +1,13 @@
+### A time-series read of the meta field scans the index over the buckets
+
+The Rust MongoDB server now builds a time-series collection's default
+`m_1_t_1` index over its buckets, keyed as mongod keys it there
+(`{meta: 1, control.min.t: 1, control.max.t: 1}`), and an `aggregate`
+filtering on the meta field reads the buckets in that index's order, 32 at
+a time and series by series, as mongod 8.2.11 does.
+
+#### Fixed
+
+- A time-series `aggregate` filtering on the meta field read the whole
+  collection at the command instead of buffering buckets from the index.
+  Fixes RM-457 (tests/backlog/test_rm_found_2026_11_09_rm348.py::test_a_time_series_read_of_the_meta_index_buffers_its_buckets)

changelog.d/rm-462-bsoncolumn-measurement.md

@@ -0,0 +1,13 @@
+### Rust MongoDB server: a time-series measurement holding BSONColumn data is refused
+
+A measurement whose data field is, or holds at any depth, a binary of
+subtype 7 is now refused as mongod 8.2.11 refuses it, instead of stored.
+
+#### Fixed
+
+- Inserting such a measurement answers `12506300`, `time-series insert
+  failed: <ns> :: caused by :: Timeseries collections cannot store
+  BSONColumn data`, and stores nothing. The meta field is not checked, as
+  mongod appends only data fields to a column.
+
+Fixes RM-462 (tests/backlog/test_rm_462.py::test_a_measurement_cannot_hold_a_bsoncolumn)

changelog.d/rm-548-ts-bucket-index-listing.md

@@ -0,0 +1,17 @@
+### Rust MongoDB server: a time-series collection's indexes list on its buckets as mongod keeps them
+
+`listIndexes` of `system.buckets.<name>` now shows each index of a
+time-series collection as mongod 8.2.11 keeps it on the buckets.
+
+#### Fixed
+
+- A TTL index on the time field is keyed `{control.min.<time>: 1}` alone,
+  not the `control.min` / `control.max` pair.
+- A partial index's filter is listed translated to the buckets: the meta
+  field's predicates on `meta` with an implicit equality spelled `$eq`, a
+  measurement's comparisons as `control.min` / `control.max` bounds, the
+  time field's without a bucket `_id` bound.
+- A partial index carries the spec as asked in `originalSpec`, which the
+  collection's own `listIndexes` does not show.
+
+Fixes RM-548 (tests/backlog/test_rm_found_2026_11_11_rm088.py::test_bucket_indexes_list_a_ttl_index_as_mongod_keeps_it)

crates/secantus-commands/src/admin.rs

@@ -591,6 +591,22 @@ pub fn create(doc: &Document, ctx: &mut CommandContext) -> HandlerResult {
         storage
             .create_index(&ctx.db_name, &coll, &name, &key, &spec)
             .map_err(command_error)?;
+        // mongod builds it over the buckets, keyed as it keys it there
+        // (`{meta: 1, control.min.t: 1, control.max.t: 1}`): a read of the
+        // meta field scans it (RM-457).
+        if let Some(layout) = crate::ts_store::layout_of(storage, &ctx.db_name, &coll) {
+            let on_buckets = crate::ts_buckets::bucket_key(&key, &layout);
+            let spec = doc! { "key": on_buckets.clone(), "name": name.clone() };
+            storage
+                .create_index(
+                    &ctx.db_name,
+                    &crate::ts_store::buckets_ns(&coll),
+                    &name,
+                    &on_buckets,
+                    &spec,
+                )
+                .map_err(command_error)?;
+        }
     }
     Ok(doc! { "ok": 1.0 })
 }
@@ -1946,6 +1962,7 @@ pub(crate) fn listed_indexes(
         ix.remove("buildSeq");
         ix.remove("binDataOrder");
         ix.remove("multikeyPaths");
+        ix.remove(TS_ORIGINAL_SPEC);
         *ix = listed_spec(std::mem::take(ix));
     }
     // The `_id` index of a collection with a default collation carries it.
@@ -2473,6 +2490,19 @@ pub fn create_indexes(doc: &Document, ctx: &mut CommandContext) -> HandlerResult
         // falsy value is dropped so it doesn't come back from listIndexes
         // (mongo-ruby-driver `hidden is false` asserts hidden isn't echoed).
         let mut spec = s.clone();
+        // A partial index on a time-series collection keeps the spec as
+        // asked, which `listIndexes` of its buckets shows as `originalSpec`
+        // (mongod 8.2.11): the fields as given, then `key` and `v` (RM-548).
+        if ts_fields.is_some() && s.contains_key("partialFilterExpression") {
+            let mut original: Document = s
+                .iter()
+                .filter(|(k, _)| k.as_str() != "key" && k.as_str() != "v")
+                .map(|(k, v)| (k.clone(), v.clone()))
+                .collect();
+            original.insert("key", key.clone());
+            original.insert("v", s.get("v").cloned().unwrap_or(Bson::Int32(2)));
+            spec.insert(TS_ORIGINAL_SPEC, original);
+        }
         for opt in ["hidden", "sparse", "unique"] {
             if spec.get(opt).is_some_and(is_falsy) {
                 spec.remove(opt);
@@ -3942,7 +3972,12 @@ fn equivalent_spec_text(spec: &Document) -> String {
 /// The options that make two index specs different indexes.
 /// Catalog bookkeeping that is not part of an index's options (and that
 /// `listIndexes` already hides from clients).
+/// The catalog's copy of a time-series partial index's spec as asked
+/// (RM-548), listed on the buckets as `originalSpec`.
+pub(crate) const TS_ORIGINAL_SPEC: &str = "timeseriesOriginalSpec";
+
 pub(crate) const INDEX_CATALOG_ONLY: &[&str] = &[
+    TS_ORIGINAL_SPEC,
     // `createIndexes`'s own mark on an index still being built (RM-187).
     BUILDING_MARK,
     "entryFormat",

crates/secantus-commands/src/agg_stream.rs

@@ -410,6 +410,33 @@ fn excludes_only(spec: &Bson) -> bool {
     matches!(spec, Bson::Document(d) if !d.is_empty() && d.values().all(zero))
 }
 
+/// A `$project` that only keeps fields by name (`{v: 1}`, `_id` left out or
+/// not): the fields mongod's `$_internalUnpackBucket` must unpack are then
+/// known, and its slot-based engine takes the pipeline.
+fn includes_only(spec: &Bson) -> bool {
+    fn kept(v: &Bson) -> bool {
+        match v {
+            Bson::Boolean(b) => *b,
+            Bson::Int32(n) => *n != 0,
+            Bson::Int64(n) => *n != 0,
+            Bson::Double(n) => *n != 0.0,
+            _ => false,
+        }
+    }
+    let Bson::Document(d) = spec else {
+        return false;
+    };
+    d.iter().any(|(k, v)| k != "_id" && kept(v))
+        && d.iter().all(|(k, v)| {
+            kept(v)
+                || (k == "_id"
+                    && matches!(
+                        v,
+                        Bson::Boolean(_) | Bson::Int32(_) | Bson::Int64(_) | Bson::Double(_)
+                    ))
+        })
+}
+
 /// The cursor of a pipeline over a time-series collection: its buckets read
 /// through the cursor stage, each unpacked as the cursor reaches it
 /// (`ts_buckets::BucketScan`, RM-348). Read whole instead: under a hint or
@@ -428,9 +455,27 @@ fn time_series(
                     .and_then(|d| d.get("$project"))
                     .is_some_and(excludes_only))
     };
+    let project = |s: &Bson| {
+        stage_name(s) == "$project"
+            && s.as_document()
+                .and_then(|d| d.get("$project"))
+                .is_some_and(includes_only)
+    };
+    // `$match` and an inclusion `$project` are the slot-based engine's: it
+    // unpacks bucket by bucket as the cursor asks, with no cursor-stage
+    // buffer (measured 8.2.11, 2026-11-15: the rest of the first bucket
+    // after a delete, 71, RM-456). An inclusion `$project` alone is read
+    // through the cursor stage's buffer like the stages above (2,303).
+    let per_bucket = ask.pipeline.iter().any(project)
+        && ask.pipeline.iter().any(|s| stage_name(s) == "$match")
+        && ask
+            .pipeline
+            .iter()
+            .all(|s| project(s) || stage_name(s) == "$match");
+    let only_projects = !ask.pipeline.is_empty() && ask.pipeline.iter().all(project);
     if ask.hint.is_some()
         || ask.collation.is_some()
-        || !ask.pipeline.iter().all(fits)
+        || !(per_bucket || only_projects || ask.pipeline.iter().all(fits))
         || ask.pipeline.iter().any(mentions_meta)
     {
         return Ok(None);
@@ -446,12 +491,16 @@ fn time_series(
         .and_then(|o| crate::timeseries::Fields::of(&o))
         .and_then(|f| f.default_index())
         .map(|(name, _)| name);
-    let own_index = storage
-        .list_indexes(ask.db, ask.coll)
-        .map_err(read_error)?
+    let listed = storage.list_indexes(ask.db, ask.coll).map_err(read_error)?;
+    let names: Vec<&str> = listed
         .iter()
         .filter_map(|spec| spec.get_str("name").ok())
-        .any(|name| name != "_id_" && Some(name) != default_index.as_deref());
+        .collect();
+    let own_index = names
+        .iter()
+        .any(|name| *name != "_id_" && Some(*name) != default_index.as_deref());
+    // A dropped default index leaves its copy over the buckets: unused.
+    let default_index = default_index.filter(|d| names.contains(&d.as_str()));
     if own_index {
         return Ok(None);
     }
@@ -460,15 +509,26 @@ fn time_series(
     else {
         return Ok(None);
     };
-    if scan.filters_meta() {
-        return Ok(None);
-    }
-    let cursor = AggStream::new(
+    // A filter on the meta field is mongod's scan of the default index
+    // over the buckets (RM-457).
+    let scan = if scan.filters_meta() {
+        match default_index {
+            Some(name) => match scan.through_index(storage, &name).map_err(read_error)? {
+                Some(scan) => scan,
+                None => return Ok(None),
+            },
+            None => return Ok(None),
+        }
+    } else {
+        scan
+    };
+    let mut cursor = AggStream::new(
         Feed::Buckets(Box::new(scan)),
-        Some(FIRST_BUFFER),
+        Some(if per_bucket { 1 } else { FIRST_BUFFER }),
         ask.pipeline.to_vec(),
         ask,
     );
+    cursor.per_bucket = per_bucket;
     Ok(Some((cursor, Vec::new())))
 }
 
@@ -705,6 +765,9 @@ struct AggStream {
     out_calls: VecDeque<u64>,
     /// The calls each stage owes the next document it passes on.
     owed: Vec<u64>,
+    /// A time-series read the slot-based engine unpacks one bucket at a
+    /// time, with no doubling cursor-stage buffer (RM-456).
+    per_bucket: bool,
 }
 
 fn stage_name(s: &Bson) -> &str {
@@ -758,6 +821,7 @@ impl AggStream {
             in_plan: 0,
             ticks: secantus_core::deadline::FRESH_TICKS,
             out_calls: VecDeque::new(),
+            per_bucket: false,
             owed: vec![0; stages_len],
         }
     }
@@ -929,7 +993,9 @@ impl AggStream {
             }
             Feed::Buckets(scan) => {
                 let n = self.buffer.unwrap_or(FIRST_BUFFER);
-                self.buffer = Some(n * 2);
+                if !self.per_bucket {
+                    self.buffer = Some(n * 2);
+                }
                 let read = scan.next_buffer(storage, n, BUFFER_BYTES)?;
                 Ok(StreamedBatch {
                     docs: read.docs,

crates/secantus-commands/src/crud.rs

@@ -1110,6 +1110,23 @@ pub fn insert(doc: &Document, ctx: &mut CommandContext) -> HandlerResult {
             }
             continue;
         }
+        if ts_fields
+            .as_ref()
+            .is_some_and(|f| crate::timeseries::holds_bson_column(d, f))
+        {
+            pre_errors.push(doc! {
+                "index": index as i32,
+                "code": 12506300,
+                "errmsg": format!(
+                    "time-series insert failed: {db}.{coll} :: caused by :: Timeseries \
+                     collections cannot store BSONColumn data"
+                ),
+            });
+            if ordered {
+                break;
+            }
+            continue;
+        }
         // Collection validator (code 121, DocumentValidationFailure). mongod
         // attaches an `errInfo` (failingDocumentId + per-operator details) that
         // drivers' errorResponse tests read; mirror it.

crates/secantus-commands/src/session.rs

@@ -952,6 +952,8 @@ fn statement(
     let mut result = {
         let mut t = txn.lock().unwrap_or_else(|e| e.into_inner());
         let handle = t.handle.as_mut().expect("handle created above").as_mut();
+        // A cause left by an earlier statement is not this one's.
+        let _ = storage.take_write_conflict_cause();
         match storage.run_in_user_transaction(handle, &mut || crate::run_handler(handler, doc, ctx))
         {
             Ok(r) => r,
@@ -971,11 +973,16 @@ fn statement(
         return result;
     }
     if name == "insert" && code == 112 {
-        // An insert conflicts on an index, and mongod names its own table for
-        // that index as the cause, or nothing.
+        // An insert conflicts on an index, and mongod names that index and
+        // its table as the cause (`WiredTigerIdIndex::_insert: ...` for the
+        // `_id` index), or nothing where the storage does not say (RM-087).
+        let cause = storage.take_write_conflict_cause().unwrap_or_default();
         result.insert(
             "errmsg",
-            "Caused by ::  :: Please retry your operation or multi-document transaction.",
+            format!(
+                "Caused by :: {cause} :: Please retry your operation or multi-document \
+                 transaction."
+            ),
         );
     }
     crate::finish_txn_statement(registry, txn, &mut result);
@@ -1247,9 +1254,11 @@ fn retryable_write(
     // The statement ids are part of the command: a malformed set is refused
     // before the session's number moves.
     let count = match name {
+        // mongod drops a null from `documents` before it counts them
+        // against `stmtIds` (RM-087, 8.2.11).
         "insert" => match (&ctx.raw_insert_documents, doc.get_array("documents")) {
             (Some(raw), _) => raw.len(),
-            (None, Ok(docs)) => docs.len(),
+            (None, Ok(docs)) => docs.iter().filter(|d| !matches!(d, Bson::Null)).count(),
             _ => 0,
         },
         "update" => doc.get_array("updates").map_or(0, Vec::len),

crates/secantus-commands/src/storage.rs

@@ -1695,6 +1695,13 @@ pub trait Storage: Send + Sync {
         Ok(f())
     }
 
+    /// What this thread's last write conflict conflicted in, as mongod names
+    /// it in a 112's cause (`WiredTigerIdIndex::_insert: index: _id_; uri:
+    /// table:<ident>`), taken so it is reported once. Default: unknown.
+    fn take_write_conflict_cause(&self) -> Option<String> {
+        None
+    }
+
     /// Has the catalog entry of `db.coll` (its indexes, its options, its
     /// existence) changed since `handle`'s snapshot was taken? mongod refuses
     /// such a transaction's write there (112). Default: never.