diff --git a/changelog.d/ftdc-diagnostic-data-archive.md b/changelog.d/ftdc-diagnostic-data-archive.md new file mode 100644 index 000000000..a2781a98e --- /dev/null +++ b/changelog.d/ftdc-diagnostic-data-archive.md @@ -0,0 +1,23 @@ +### The Rust MongoDB server writes its FTDC samples to `diagnostic.data` + +Every full-time diagnostic data capture sample is now written to +`/diagnostic.data` in mongod's own archive format, so mongod's FTDC +decoders read it: `metrics.-00000` files that open with the +`buildInfo` / `getCmdLineOpts` / `hostInfo` metadata, then zlib-compressed +metric chunks of a reference sample and its delta-encoded successors, with +`metrics.interim` carrying the chunk in progress into the next start. + +#### Fixed + +- `diagnosticDataCollectionDirectoryPath` answers the directory, and a change + to it is refused 220 `FTDCPathAlreadySet`, as on mongod 8.2.11. +- `diagnosticDataCollectionDirectorySizeMB`, `...FileSizeMB`, + `...SamplesPerChunk` and `...SamplesPerInterimUpdate` are registered with + mongod's minimums and its file-versus-directory size check; files rotate at + the file size and the oldest are removed past the directory size. + +Fixes RM-417 (tests/backlog/test_rm_219_diag_query_settings.py::test_samples_are_written_to_diagnostic_data) + +#### Filed + +- RM-690: the archive writes no `type: 2` periodic metadata. diff --git a/changelog.d/rm-418-debug-query-shape.md b/changelog.d/rm-418-debug-query-shape.md new file mode 100644 index 000000000..1706e9689 --- /dev/null +++ b/changelog.d/rm-418-debug-query-shape.md @@ -0,0 +1,11 @@ +### Rust MongoDB server: `$querySettings` gives `debugQueryShape` + +`{$querySettings: {showDebugQueryShape: true}}` now adds each setting's +`debugQueryShape` (its query's shape with each literal a debug type +string, `{x: {$eq: "?number"}}`), as mongod 8.2 does. Before, the option +was parsed and ignored. + +#### Fixed + +- `showDebugQueryShape` gave no `debugQueryShape`. + Fixes RM-418 in part (tests/backlog/test_rm_219_diag_query_settings.py::test_show_debug_query_shape) diff --git a/changelog.d/rm-540-applyops-u.md b/changelog.d/rm-540-applyops-u.md new file mode 100644 index 000000000..196167ae2 --- /dev/null +++ b/changelog.d/rm-540-applyops-u.md @@ -0,0 +1,15 @@ +### Rust MongoDB server: `applyOps` `u` entries answer as mongod does + +`applyOps` now leaves a `u` replacement of a missing document unapplied when +the entry has no `b: true`, and refuses documents past 16 MB with mongod +8.2.11's codes. + +#### Fixed + +- A `u` entry replacing a document that does not exist, without `b: true`, + is `false` in `results` (the command fails 8); it was inserted. +- A `u` upsert of a document past 16 MB is 17420 "Document to upsert is + larger than 16777216" (was 10334), and an `i` entry replacing an existing + document with one past 16 MB is 17419 (was 10334). + +Fixes RM-540 (tests/backlog/test_rm_219_diag_query_settings.py::test_apply_ops_update_entries) diff --git a/changelog.d/rm-541-int-param-double-bounds.md b/changelog.d/rm-541-int-param-double-bounds.md new file mode 100644 index 000000000..4303202b9 --- /dev/null +++ b/changelog.d/rm-541-int-param-double-bounds.md @@ -0,0 +1,9 @@ +### Rust MongoDB server: an int parameter bounds a double before truncating it + +`setParameter` now refuses `2147483647.9` and `-2147483648.5` for an int +parameter with mongod 8.2.11's "Out of bounds coercing to integral value", +instead of truncating them into range and storing them. + +#### Fixed + +- Fixes RM-541 (tests/backlog/test_rm_found_2026_11_10_rm417.py::test_an_int_parameter_bounds_the_double_before_truncating) diff --git a/changelog.d/rm-543-sbe-find-in-projection.md b/changelog.d/rm-543-sbe-find-in-projection.md new file mode 100644 index 000000000..09e936682 --- /dev/null +++ b/changelog.d/rm-543-sbe-find-in-projection.md @@ -0,0 +1,17 @@ +### Rust MongoDB server: an `$in` and a simple projection explain as SBE under `queryFramework: "sbe"` + +A `find` that a query setting puts in SBE now explains as the slot program +mongod 8.2.11 runs when its filter holds an `$in` or it projects top-level +fields. Before, it was explained in the classic shape. + +#### Fixed + +- An `$in` is `isMember` against an `InList` that the plan's environment + binds, holding the values in the command's order. A simple projection is + `makeBsonObj` (closed for an inclusion, open for an exclusion), or + `newBsonObj` when it keeps one field. It sits above a skip and below a + limit. The `queryPlan`, the slot program and `executionStats` match + mongod's. A sorted or index-served `find` is still explained classically + (RM-543 stays open for those). + +Fixes RM-543 in part (tests/backlog/test_rm_found_2026_11_10_rm418.py::test_sbe_find_in_and_projection_are_mongods). diff --git a/crates/Cargo.lock b/crates/Cargo.lock index a28c1b7f6..52b6e5740 100644 --- a/crates/Cargo.lock +++ b/crates/Cargo.lock @@ -1757,6 +1757,7 @@ version = "0.5.3-beta.196" dependencies = [ "base64", "bson", + "flate2", "hmac 0.12.1", "libc", "log", @@ -1767,6 +1768,7 @@ dependencies = [ "secantus-core", "secantus-wire", "sha2 0.10.9", + "tempfile", ] [[package]] diff --git a/crates/secantus-commands/Cargo.toml b/crates/secantus-commands/Cargo.toml index eb2dee75c..0c9a69c80 100644 --- a/crates/secantus-commands/Cargo.toml +++ b/crates/secantus-commands/Cargo.toml @@ -45,6 +45,8 @@ md-5 = "0.10" base64 = "0.22" # `$queryStats`' `transformIdentifiers`: each identifier as its HMAC-SHA-256. hmac = "0.12" +# The FTDC archive's metric chunks are zlib, as mongod writes them. +flate2 = "1" # `dbHash` names the machine as mongod does (`hostname:port`). [target.'cfg(unix)'.dependencies] @@ -56,3 +58,4 @@ libc = "0.2" hmac = "0.12" sha2 = "0.10" pbkdf2 = "0.12" +tempfile = "3" diff --git a/crates/secantus-commands/src/apply_ops.rs b/crates/secantus-commands/src/apply_ops.rs index 109a08ab5..011e0a77f 100644 --- a/crates/secantus-commands/src/apply_ops.rs +++ b/crates/secantus-commands/src/apply_ops.rs @@ -463,7 +463,10 @@ fn apply_entry( let post = match (diff, current) { (Some(diff), Some(pre)) => apply_diff(pre, diff)?, (Some(diff), None) if upsert => apply_diff(doc! {"_id": id.clone()}, diff)?, - (None, Some(_)) | (None, None) if upsert || o.get("_id") == Some(&id) => { + // A replacement of a missing document is applied only as an + // upsert: mongod 8.2.11 leaves it `false` in `results`. + (None, Some(_)) if o.get("_id") == Some(&id) => o.clone(), + (None, None) if upsert => { if o.get("_id") != Some(&id) { return Err(error(13596, "in Collection::updateDocument _id mismatch")); } @@ -526,6 +529,9 @@ fn apply_entry( } } +/// The largest document a client may write (mongod's `BSONObjMaxUserSize`). +const BSON_USER_MAX: usize = 16 * 1024 * 1024; + fn is_timeseries(storage: &dyn crate::Storage, db: &str, coll: &str) -> bool { storage .get_collection_options(db, coll) @@ -581,11 +587,26 @@ fn write_document( let filter = doc! {"_id": id.clone()}; let outcome = storage .update_matching(db, coll, &filter, &doc, false, false) - .map_err(crate::util::command_error)?; + .map_err(|e| { + // A replacement past 16 MiB is 17419 under oplog application on + // mongod 8.2.11, with the update path's message. + let e = crate::util::command_error(e); + if e.code == 10334 && e.errmsg.starts_with("Resulting document after update") { + error(17419, e.errmsg) + } else { + e + } + })?; if outcome.matched > 0 { return Ok(Outcome::Applied); } let bytes = crate::bsondoc::to_vec(&doc).map_err(|e| error(1, e.to_string()))?; + if !insert_entry && bytes.len() > BSON_USER_MAX { + return Err(error( + 17420, + format!("Document to upsert is larger than {BSON_USER_MAX}"), + )); + } let (_, errors) = if insert_entry { storage.insert_from_oplog_application(db, coll, vec![bytes], true) } else { diff --git a/crates/secantus-commands/src/explain/mod.rs b/crates/secantus-commands/src/explain/mod.rs index f9d28933a..5459d6c46 100644 --- a/crates/secantus-commands/src/explain/mod.rs +++ b/crates/secantus-commands/src/explain/mod.rs @@ -1957,8 +1957,9 @@ fn reply_for(r: &Request, verbosity: Verbosity, ctx: &mut CommandContext) -> Han } } // A `find` mongod runs in SBE (RM-418): over a forward collection scan - // with no sort, projection, hint or collation, its scan, filter, skip - // and limit; any other is still explained in the classic shape. + // with no sort, hint or collation, its scan, filter, skip, limit and a + // simple projection (RM-543); any other is still explained in the + // classic shape. if r.pipeline.is_empty() && front_runs_in_sbe(storage, &db, s, collation.as_ref(), &[]) { let built = if !t.exists { Some(sbe::eof_plan()) @@ -1971,14 +1972,13 @@ fn reply_for(r: &Request, verbosity: Verbosity, ctx: &mut CommandContext) -> Han } ); let simple = s.sort.as_ref().is_none_or(Document::is_empty) - && s.projection.as_ref().is_none_or(Document::is_empty) && s.hint.is_none() && collation.is_none() && s.min_max.is_none() && !s.return_key && !s.show_record_id; let parsed = parsed::parsed_query(&s.filter); - let filter = sbe::Filter::parse(&parsed); + let filter = sbe::Filter::parse_find(&parsed, &s.filter); let on = sbe::Front { parsed, filter, @@ -1988,7 +1988,7 @@ fn reply_for(r: &Request, verbosity: Verbosity, ctx: &mut CommandContext) -> Han let filtered = s.filter.is_empty() || on.filter.is_some(); let uuid = crate::admin::collection_uuid_text(storage, &db, &s.coll); (scan && simple && filtered) - .then(|| sbe::find_plan(&uuid, &on)) + .then(|| sbe::find_plan(&uuid, &on, s.projection.as_ref())) .flatten() }; if let Some(built) = built { diff --git a/crates/secantus-commands/src/explain/sbe/expr.rs b/crates/secantus-commands/src/explain/sbe/expr.rs index 942236d4e..1b4d244a7 100644 --- a/crates/secantus-commands/src/explain/sbe/expr.rs +++ b/crates/secantus-commands/src/explain/sbe/expr.rs @@ -54,6 +54,12 @@ pub(crate) enum Expr { TypeMatch(Box, u32), /// `ENumericConvert` to int32: `convert ( e, int32)`. ConvertInt32(Box), + /// A slot the plan's environment binds to an `InList` (an `$in`'s + /// values, in the order the command gave them): prints `sN`. + InList(Slot, Vec), + /// `makeBsonObj`'s spec: the fields it keeps (`Closed`, the rest + /// dropped) or drops (`Open`, the rest kept), in the projection's order. + ObjSpec { fields: Vec, open: bool }, } /// The mask `typeMatch` and `fillType` take for null and undefined: @@ -61,6 +67,36 @@ pub(crate) enum Expr { pub(crate) const NULLISH_MASK: u32 = 1088; impl Expr { + /// The `InList`s the expression reads, in the order it prints them. + pub fn in_lists(&self) -> Vec<(Slot, Vec)> { + let mut out = Vec::new(); + self.collect_lists(&mut out); + out + } + + fn collect_lists(&self, out: &mut Vec<(Slot, Vec)>) { + match self { + Expr::InList(s, values) => out.push((*s, values.clone())), + Expr::Func(_, args) => args.iter().for_each(|a| a.collect_lists(out)), + Expr::Bin(_, l, r) => { + l.collect_lists(out); + r.collect_lists(out); + } + Expr::If(c, t, e) => { + c.collect_lists(out); + t.collect_lists(out); + e.collect_lists(out); + } + Expr::Not(e) | Expr::TypeMatch(e, _) | Expr::ConvertInt32(e) => e.collect_lists(out), + Expr::Lambda { body, .. } => body.collect_lists(out), + Expr::Slot(_) + | Expr::Local { .. } + | Expr::Const(_) + | Expr::Int64(_) + | Expr::ObjSpec { .. } => {} + } + } + pub fn null() -> Expr { Expr::Const(Some(Bson::Null)) } @@ -326,6 +362,9 @@ impl Expr { } _ => None, }, + Expr::InList(_, values) => Some(Bson::Array(values.clone())), + // Read only by the `makeBsonObj` it is the spec of. + Expr::ObjSpec { .. } => None, Expr::Func(name, args) => self.call(name, args, env), } } @@ -375,6 +414,39 @@ impl Expr { let vals: Vec = args.iter().map(|a| a.eval(env)).collect(); new_obj(&vals) } + // `isMember(v, inList)`: `v` equals one of the list's values + // under BSON comparison; Nothing is a member of nothing. + "isMember" => { + let v = args[0].eval(env); + let list = args[1].eval(env); + match (v, list) { + (Some(v), Some(Bson::Array(list))) => { + Some(Bson::Boolean(list.iter().any(|m| { + compare(&Some(m.clone()), &Some(v.clone())) == Ordering::Equal + }))) + } + (None, Some(_)) => Some(Bson::Boolean(false)), + _ => None, + } + } + // `makeBsonObj(spec, root)`: the root's fields, kept or dropped + // by the spec, in the root's order; Nothing over a non-object + // (`RetNothing`) for a closed spec, the input for an open one + // (`RetInput`). + "makeBsonObj" => { + let Expr::ObjSpec { fields, open } = &args[0] else { + return None; + }; + match args[1].eval(env) { + Some(Bson::Document(d)) => Some(Bson::Document( + d.into_iter() + .filter(|(k, _)| fields.contains(k) != *open) + .collect(), + )), + other if *open => other, + _ => None, + } + } "isNumber" => args[0].eval(env).map(|v| Bson::Boolean(is_number(&v))), "fillType" => { let v = args[0].eval(env); diff --git a/crates/secantus-commands/src/explain/sbe/filter.rs b/crates/secantus-commands/src/explain/sbe/filter.rs index fe6464417..24583c0fc 100644 --- a/crates/secantus-commands/src/explain/sbe/filter.rs +++ b/crates/secantus-commands/src/explain/sbe/filter.rs @@ -14,19 +14,29 @@ //! `l6.0`, `d.e` `l3.0` and `l4.0`). //! //! Built here: `$eq`, `$lt`, `$lte`, `$gt` and `$gte` of a number (not NaN), -//! a string or a boolean, on a path of named fields, and `$and`s of those. +//! a string or a boolean, on a path of named fields, and `$and`s of those; +//! for a `find` (`Filter::parse_find`), also an `$in` of such values but +//! null: `isMember` against an `InList` the plan's environment binds, its +//! values in the command's own order (the `parsedQuery` sorts them). //! Anything else returns `None` and the pipeline is explained classically. use super::expr::{BinOp, Expr, Slot}; use bson::{Bson, Document}; use std::collections::BTreeSet; +/// What a leaf tests its path's values with. +#[derive(Clone, Debug)] +enum Test { + Compare(BinOp, Bson), + /// `$in`: the values as the command gave them. + In(Vec), +} + /// One comparison of the filter. #[derive(Clone, Debug)] struct Leaf { path: Vec, - op: BinOp, - value: Bson, + test: Test, } /// A filter, ready to be given its slots. @@ -57,7 +67,7 @@ fn comparable(v: &Bson) -> bool { } } -fn leaf(clause: &Document) -> Option { +fn leaf(clause: &Document, raw: Option<&Document>) -> Option { let (path, cond) = clause.iter().next().filter(|_| clause.len() == 1)?; if path.starts_with('$') { return None; @@ -73,24 +83,65 @@ fn leaf(clause: &Document) -> Option { return None; }; let (name, value) = cond.iter().next().filter(|_| cond.len() == 1)?; + if name == "$in" { + let values = raw_in(raw?, path)?; + let printable = values + .iter() + .all(|v| comparable(v) && super::print::list_value(v).is_some()); + return printable.then_some(Leaf { + path: parts, + test: Test::In(values), + }); + } let op = op_of(name)?; comparable(value).then(|| Leaf { path: parts, - op, - value: value.clone(), + test: Test::Compare(op, value.clone()), }) } +/// The one `$in` on `path` in the command's filter `raw`, at its top level +/// or in a top-level `$and`; `None` when there is not exactly one. +fn raw_in(raw: &Document, path: &str) -> Option> { + let mut found = Vec::new(); + let mut look = |d: &Document| { + if let Some(Bson::Document(cond)) = d.get(path) { + if let Some(Bson::Array(values)) = cond.get("$in") { + found.push(values.clone()); + } + } + }; + look(raw); + if let Some(Bson::Array(clauses)) = raw.get("$and") { + for c in clauses { + if let Bson::Document(c) = c { + look(c); + } + } + } + (found.len() == 1).then(|| found.remove(0)) +} + impl Filter { /// The filter of `parsed` (a `parsedQuery`); `None` when it is empty or /// outside what is built here. pub fn parse(parsed: &Document) -> Option { + Self::parse_with(parsed, None) + } + + /// The filter of a `find` whose command filter is `raw` and whose + /// `parsedQuery` is `parsed`: an `$in` is built too. + pub fn parse_find(parsed: &Document, raw: &Document) -> Option { + Self::parse_with(parsed, Some(raw)) + } + + fn parse_with(parsed: &Document, raw: Option<&Document>) -> Option { let leaves = match parsed.get("$and") { Some(Bson::Array(clauses)) if parsed.len() == 1 => clauses .iter() - .map(|c| leaf(c.as_document()?)) + .map(|c| leaf(c.as_document()?, raw)) .collect::>>()?, - _ => vec![leaf(parsed)?], + _ => vec![leaf(parsed, raw)?], }; Some(Filter { leaves }) } @@ -109,13 +160,33 @@ impl Filter { self.leaves.iter().map(|l| l.path.len() as u32).sum() } - /// The program, reading each top-level field from `slot_of`. - pub fn program(&self, slot_of: &dyn Fn(&str) -> Option) -> Option { + /// The program, reading each top-level field from `slot_of`; each + /// `$in`'s list takes the next slot of `slots`, in the program's order. + pub fn program( + &self, + slot_of: &dyn Fn(&str) -> Option, + slots: &mut dyn FnMut() -> Slot, + ) -> Option { let mut frame = self.lambdas(); let mut built = Vec::new(); for l in &self.leaves { let top = Expr::Slot(slot_of(&l.path[0])?); - built.push(traverse(top, l, 0, &mut frame)); + let test: Box Expr> = match &l.test { + Test::In(values) => { + let list = Expr::InList(slots(), values.clone()); + Box::new(move |v| Expr::func("isMember", vec![v, list.clone()])) + } + Test::Compare(op, value) => { + let (op, value) = (*op, value.clone()); + Box::new(move |v| { + Expr::fill_empty( + Expr::bin(op, v, Expr::Const(Some(value.clone()))), + Expr::Const(Some(Bson::Boolean(false))), + ) + }) + } + }; + built.push(traverse(top, l, &*test, 0, &mut frame)); } let mut out = built.pop()?; while let Some(prev) = built.pop() { @@ -127,7 +198,13 @@ impl Filter { /// `traverseF` over `input`, the field `leaf.path[depth]` of the level /// above; frames are taken outermost first. -fn traverse(input: Expr, leaf: &Leaf, depth: usize, frame: &mut u32) -> Expr { +fn traverse( + input: Expr, + leaf: &Leaf, + test: &dyn Fn(Expr) -> Expr, + depth: usize, + frame: &mut u32, +) -> Expr { *frame += 1; let mine = *frame; let local = Expr::Local { @@ -135,10 +212,7 @@ fn traverse(input: Expr, leaf: &Leaf, depth: usize, frame: &mut u32) -> Expr { moved: true, }; let body = if depth + 1 == leaf.path.len() { - Expr::fill_empty( - Expr::bin(leaf.op, local, Expr::Const(Some(leaf.value.clone()))), - Expr::Const(Some(Bson::Boolean(false))), - ) + test(local) } else { let field = Expr::func( "getField", @@ -147,7 +221,7 @@ fn traverse(input: Expr, leaf: &Leaf, depth: usize, frame: &mut u32) -> Expr { Expr::Const(Some(Bson::String(leaf.path[depth + 1].clone()))), ], ); - traverse(field, leaf, depth + 1, frame) + traverse(field, leaf, test, depth + 1, frame) }; Expr::func( "traverseF", @@ -171,7 +245,7 @@ mod tests { fn text(parsed: bson::Document, slots: &[(&str, u32)]) -> String { let f = Filter::parse(&parsed).unwrap(); let slot = |n: &str| slots.iter().find(|(k, _)| *k == n).map(|(_, s)| *s); - expression(&f.program(&slot).unwrap()) + expression(&f.program(&slot, &mut || 9).unwrap()) } /// The filters mongod 8.2.11 prints, measured 2026-11-08. diff --git a/crates/secantus-commands/src/explain/sbe/find.rs b/crates/secantus-commands/src/explain/sbe/find.rs index 22a81e6eb..64b65b370 100644 --- a/crates/secantus-commands/src/explain/sbe/find.rs +++ b/crates/secantus-commands/src/explain/sbe/find.rs @@ -4,25 +4,189 @@ //! in the classic engine (RM-418). //! //! Built here: the collection scan, its filter (`filter.rs`), skip and -//! limit, as `buildCollScan` and `buildLimitSkip` build them; the plan's -//! result is the scanned record itself. A sorted, projected or -//! index-served `find` is not built (`None`). +//! limit, as `buildCollScan` and `buildLimitSkip` build them, and a simple +//! projection over them (`buildProjectionSimple`, RM-543): an inclusion or +//! exclusion of top-level fields. The plan's result is the scanned record, +//! or the projected one. A sorted or index-served `find` is not built +//! (`None`), nor a projection of a path, a computed field or a `$` operator. +use super::expr::Expr; use super::group::{scan, Front, Scanned, Slots}; +use super::stage::Stage; use super::Plan; +use bson::{doc, Bson, Document}; +use std::collections::BTreeSet; -pub(super) fn build(uuid: &str, front: &Front) -> Option { +/// A projection of top-level fields, as the planner keeps it. +struct Projection { + /// The fields kept (an inclusion) or dropped (an exclusion), in the + /// projection's order; an inclusion keeps `_id` last unless it names it. + fields: Vec, + inclusion: bool, + /// The `PROJECTION_SIMPLE` node's `transformBy`. + transform_by: Document, +} + +fn truth(v: &Bson) -> Option { + Some(match v { + Bson::Boolean(b) => *b, + Bson::Int32(n) => *n != 0, + Bson::Int64(n) => *n != 0, + Bson::Double(d) if !d.is_nan() => *d != 0.0, + _ => return None, + }) +} + +fn projection(spec: &Document) -> Option { + let mut named: Vec<(String, bool)> = Vec::new(); + for (k, v) in spec { + if k.is_empty() || k.starts_with('$') || k.contains('.') { + return None; + } + named.push((k.clone(), truth(v)?)); + } + let id = named.iter().find(|(k, _)| k == "_id").map(|(_, b)| *b); + let others: Vec = named + .iter() + .filter(|(k, _)| k != "_id") + .map(|(_, b)| *b) + .collect(); + let inclusion = match (others.first(), id) { + (Some(&first), _) => { + if others.iter().any(|b| *b != first) { + return None; + } + first + } + (None, Some(b)) => b, + (None, None) => return None, + }; + // An exclusion that keeps `_id` by name: not measured. + if !inclusion && id == Some(true) { + return None; + } + let mut transform_by: Document = named + .iter() + .map(|(k, b)| (k.clone(), Bson::Boolean(*b))) + .collect(); + let mut fields: Vec = named + .iter() + .filter(|(_, b)| *b == inclusion) + .map(|(k, _)| k.clone()) + .collect(); + if inclusion && id.is_none() { + transform_by.insert("_id", true); + fields.push("_id".to_string()); + } + Some(Projection { + fields, + inclusion, + transform_by, + }) +} + +/// The plan for the `find` whose query is `front` and whose projection is +/// `spec` (`None` or empty: none), over the collection whose UUID is `uuid`. +pub(super) fn build(uuid: &str, front: &Front, spec: Option<&Document>) -> Option { let mut slots = Slots(0); + let Some(spec) = spec.filter(|p| !p.is_empty()) else { + let Scanned { + root, + record, + env, + query_plan, + .. + } = scan(front, uuid, Default::default(), &mut slots)?; + return Some(Plan { + root, + result: record, + env, + query_plan, + eof: false, + }); + }; + let p = projection(spec)?; + // A projection under a skip and a limit both: not measured. + if front.skip > 0 && front.limit > 0 { + return None; + } + // One field kept: `newBsonObj` of the field the scan reads. + let single = p.inclusion && p.fields.len() == 1; + let read: BTreeSet = if single { + p.fields.iter().cloned().collect() + } else { + BTreeSet::new() + }; + // The projection sits over the skip and under the limit: the scan + // builds the skip, the limit goes on top here. + let under = Front { + parsed: front.parsed.clone(), + filter: front.filter.clone(), + skip: front.skip, + limit: 0, + }; let Scanned { root, record, - env, + fields, + mut env, query_plan, - .. - } = scan(front, uuid, Default::default(), &mut slots)?; + } = scan(&under, uuid, read, &mut slots)?; + let value = if single { + let field = &p.fields[0]; + let slot = fields.iter().find(|(n, _)| n == field)?.1; + Expr::func( + "newBsonObj", + vec![ + Expr::Const(Some(Bson::String(field.clone()))), + Expr::Slot(slot), + ], + ) + } else { + Expr::func( + "makeBsonObj", + vec![ + Expr::ObjSpec { + fields: p.fields.clone(), + open: !p.inclusion, + }, + Expr::Slot(record), + ], + ) + }; + let node = under.nodes() + 1; + let result = slots.next(); + let mut root = Stage::Project { + node, + binds: vec![(result, value)], + input: Box::new(root), + }; + let mut query_plan = doc! { + "stage": "PROJECTION_SIMPLE", + "planNodeId": node as i32, + "transformBy": p.transform_by, + "inputStage": query_plan, + }; + if front.limit > 0 { + let limit = slots.next(); + let at = (limit, front.limit); + env.push((limit, front.limit.to_string())); + root = Stage::LimitSkip { + node: node + 1, + limit: Some(at), + skip: None, + input: Box::new(root), + }; + query_plan = doc! { + "stage": "LIMIT", + "planNodeId": (node + 1) as i32, + "limitAmount": super::number(front.limit), + "inputStage": query_plan, + }; + } Some(Plan { root, - result: record, + result, env, query_plan, eof: false, @@ -45,7 +209,11 @@ mod tests { } fn slot_based(f: &Front) -> (String, String) { - let plan = build("u", f).expect("built"); + projected(f, None) + } + + fn projected(f: &Front, p: Option) -> (String, String) { + let plan = build("u", f, p.as_ref()).expect("built"); let sbp = plan.winning_plan(false); let sbp = sbp.get_document("slotBasedPlan").unwrap(); ( @@ -101,7 +269,7 @@ mod tests { #[test] fn a_limit_stops_what_is_under_it() { let docs: Vec = (0..5).map(|i| doc! { "x": i }).collect(); - let plan = build("u", &front(doc! { "x": { "$gt": 0 } }, 0, 2)).unwrap(); + let plan = build("u", &front(doc! { "x": { "$gt": 0 } }, 0, 2), None).unwrap(); let (stats, n) = plan.execution_stats(&docs, crate::explain::sbe::Pull::Find, false); assert_eq!(n, 2); let top = stats.get_document("executionStages").unwrap(); @@ -115,4 +283,166 @@ mod tests { assert_eq!(scan.get_i32("isEOF"), Ok(0)); assert_eq!(stats.get_i32("totalDocsExamined"), Ok(3)); } + + fn find(raw: Document, parsed: Document, skip: i64, limit: i64) -> Front { + Front { + filter: crate::explain::sbe::Filter::parse_find(&parsed, &raw), + parsed, + skip, + limit, + } + } + + const SCAN_X: &str = "[1] scan s1 s2 none none none none none none [s3 = x] @\"u\" true false "; + + /// An `$in`'s `InList`, bound after the scan's fields and before a + /// limit, in the command's order (measured 8.2.11, 2026-11-14). + #[test] + fn in_plans_are_mongods() { + let member = |l: u32, s: u32, list: u32| { + format!("traverseF(s{s}, lambda(l{l}.0) {{ isMember(move(l{l}.0), s{list}) }}, false)") + }; + assert_eq!( + slot_based(&find( + doc! { "x": { "$in": [2, 1, 1] } }, + doc! { "x": { "$in": [1, 2] } }, + 0, + 0 + )), + ( + "$$RESULT=s1 env: { s4 = InList(2 1 1) }".to_string(), + format!("[1] filter {{{}}} \n{SCAN_X}", member(2, 3, 4)) + ) + ); + assert_eq!( + slot_based(&find( + doc! { "x": { "$in": [2.0, 1.5, 3_i64, -0.0, 1e20, "a b", true] } }, + doc! { "x": { "$in": [1.5, 2.0, 3_i64, -0.0, 1e20, "a b", true] } }, + 0, + 2 + )) + .0, + "$$RESULT=s1 env: { s4 = InList(2.0 1.5 3 -0.0 1e+20 \"a b\" true), s5 = 2 }" + ); + let both = slot_based(&find( + doc! { "x": { "$in": [1, 2] }, "y": { "$in": [3, 4] } }, + doc! { "$and": [{ "x": { "$in": [1, 2] } }, { "y": { "$in": [3, 4] } }] }, + 0, + 0, + )); + assert_eq!( + both.0, + "$$RESULT=s1 env: { s5 = InList(1 2), s6 = InList(3 4) }" + ); + assert!(both.1.starts_with(&format!( + "[1] filter {{({} && {})}}", + member(3, 3, 5), + member(4, 4, 6) + ))); + // A merged skip and limit: the skip's slot first, the list's next. + assert_eq!( + slot_based(&find( + doc! { "x": { "$in": [1, 2] } }, + doc! { "x": { "$in": [1, 2] } }, + 1, + 2 + )) + .0, + "$$RESULT=s2 env: { s1 = 1, s5 = InList(1 2), s6 = 2 }" + ); + // A null in the list takes another program: not built. + assert!(crate::explain::sbe::Filter::parse_find( + &doc! { "x": { "$in": [null, 1] } }, + &doc! { "x": { "$in": [null, 1] } } + ) + .is_none()); + } + + /// The projections mongod 8.2.11 builds over the scan (2026-11-14). + #[test] + fn projection_plans_are_mongods() { + let x1 = || find(doc! { "x": 1 }, doc! { "x": { "$eq": 1 } }, 0, 0); + let filter_at = |s: u32| { + format!("[1] filter {{traverseF(s{s}, lambda(l2.0) {{ ((move(l2.0) == 1) ?: false) }}, false)}} \n") + }; + let filter = filter_at(3); + for (spec, program) in [ + ( + doc! { "y": 1 }, + "makeBsonObj(MakeObjSpec([y, _id], Closed, RetNothing, 0), s1)", + ), + ( + doc! { "_id": 1, "z": 1, "y": 1 }, + "makeBsonObj(MakeObjSpec([_id, z, y], Closed, RetNothing, 0), s1)", + ), + ( + doc! { "_id": 0, "z": 1, "y": 1 }, + "makeBsonObj(MakeObjSpec([z, y], Closed, RetNothing, 0), s1)", + ), + ( + doc! { "y": 0, "_id": 0 }, + "makeBsonObj(MakeObjSpec([y, _id], Open, RetInput, 0), s1)", + ), + ( + doc! { "_id": 0 }, + "makeBsonObj(MakeObjSpec([_id], Open, RetInput, 0), s1)", + ), + (doc! { "_id": 0, "x": 1 }, "newBsonObj(\"x\", s3)"), + ] { + assert_eq!( + projected(&x1(), Some(spec.clone())), + ( + "$$RESULT=s4 env: { }".to_string(), + format!("[2] project [s4 = {program}] \n{filter}{SCAN_X}") + ), + "{spec}" + ); + } + // One field of its own: the scan reads it before the filter's. + assert_eq!( + projected(&x1(), Some(doc! { "_id": 0, "y": 1 })), + ( + "$$RESULT=s5 env: { }".to_string(), + format!( + "[2] project [s5 = newBsonObj(\"y\", s3)] \n{}\ + [1] scan s1 s2 none none none none none none [s3 = y, s4 = x] @\"u\" true false ", + filter_at(4) + ) + ) + ); + let closed = "makeBsonObj(MakeObjSpec([y, _id], Closed, RetNothing, 0), s1)"; + // Under a limit, over a skip. + let limited = find(doc! { "x": 1 }, doc! { "x": { "$eq": 1 } }, 0, 2); + assert_eq!( + projected(&limited, Some(doc! { "y": 1 })), + ( + "$$RESULT=s4 env: { s5 = 2 }".to_string(), + format!("[3] limit s5 \n[2] project [s4 = {closed}] \n{filter}{SCAN_X}") + ) + ); + let skipped = find(doc! { "x": 1 }, doc! { "x": { "$eq": 1 } }, 1, 0); + assert_eq!( + projected(&skipped, Some(doc! { "y": 1 })), + ( + "$$RESULT=s5 env: { s4 = 1 }".to_string(), + format!("[3] project [s5 = {closed}] \n[2] limitskip none s4 \n{filter}{SCAN_X}") + ) + ); + let plan = build("u", &x1(), Some(&doc! { "_id": 0, "y": 1 })).unwrap(); + let qp = plan.winning_plan(false); + let qp = qp.get_document("queryPlan").unwrap(); + assert_eq!(qp.get_str("stage"), Ok("PROJECTION_SIMPLE")); + assert_eq!( + qp.get_document("transformBy").unwrap(), + &doc! { "_id": false, "y": true } + ); + // Not built: a path, a computed field, a mixed spec. + for spec in [ + doc! { "d.e": 1 }, + doc! { "y": "$x" }, + doc! { "y": 1, "z": 0 }, + ] { + assert!(build("u", &x1(), Some(&spec)).is_none(), "{spec}"); + } + } } diff --git a/crates/secantus-commands/src/explain/sbe/group.rs b/crates/secantus-commands/src/explain/sbe/group.rs index 4e15fcb58..3b1ff22bd 100644 --- a/crates/secantus-commands/src/explain/sbe/group.rs +++ b/crates/secantus-commands/src/explain/sbe/group.rs @@ -189,8 +189,9 @@ pub(super) struct Scanned { pub record: Slot, /// The top-level fields the scan reads into slots. pub fields: Vec<(String, Slot)>, - /// The skip's and the limit's constants. - pub env: Vec<(Slot, i64)>, + /// The constants the plan's environment binds: each `$in`'s list, the + /// skip's and the limit's amounts, as the environment prints them. + pub env: Vec<(Slot, String)>, pub query_plan: Document, } @@ -227,14 +228,18 @@ pub(super) fn scan( fields: scan_fields.clone(), uuid: uuid.to_string(), }; + let mut env: Vec<(Slot, String)> = Vec::new(); if let Some(f) = &front.filter { + let predicate = f.program(&field_slot, &mut || slots.next())?; + for (slot, values) in predicate.in_lists() { + env.push((slot, super::print::in_list(&values)?)); + } root = Stage::Filter { node: 1, - predicate: f.program(&field_slot)?, + predicate, input: Box::new(root), }; } - let mut env: Vec<(Slot, i64)> = Vec::new(); let mut query_plan = doc! { "stage": "COLLSCAN", "planNodeId": 1, @@ -262,8 +267,8 @@ pub(super) fn scan( let limit_slot = (limit > 0).then(|| slots.next()); let skip_at = skip_slot.map(|s| (s, skip)); let limit_at = limit_slot.map(|s| (s, limit)); - env.extend(skip_at); - env.extend(limit_at); + env.extend(skip_at.map(|(s, n)| (s, n.to_string()))); + env.extend(limit_at.map(|(s, n)| (s, n.to_string()))); env.sort(); root = Stage::LimitSkip { node: front_nodes, diff --git a/crates/secantus-commands/src/explain/sbe/mod.rs b/crates/secantus-commands/src/explain/sbe/mod.rs index 7bd813cdd..0d838bf80 100644 --- a/crates/secantus-commands/src/explain/sbe/mod.rs +++ b/crates/secantus-commands/src/explain/sbe/mod.rs @@ -15,7 +15,7 @@ //! `$sum`, `$count`, `$avg`, `$first`, `$last`, `$min`, `$max`, `$push` and //! `$addToSet`. And a `find` a `queryFramework: "sbe"` query setting puts //! in SBE (RM-418, `find.rs`): the same collection scan, filter, skip and -//! limit, with nothing over them. Everything else SBE runs is still +//! limit, with a simple projection over them. Everything else SBE runs is still //! explained in the classic shape (`tasks/backlog-rust-mongodb.md`, RM-071, //! RM-543). @@ -36,8 +36,9 @@ use stage::{Counted, Stage}; pub(crate) struct Plan { root: Stage, result: Slot, - /// The constants the plan reads from slots: its skip and limit. - env: Vec<(Slot, i64)>, + /// The constants the plan reads from slots (its `$in` lists, skip and + /// limit), as its environment prints them. + env: Vec<(Slot, String)>, /// The `queryPlan`: the query solution the stages were built from. query_plan: Document, /// The collection does not exist: `limit 0` over `coscan`. @@ -52,11 +53,11 @@ pub(crate) fn group_plan(group: &Document, uuid: &str, last: bool, front: &Front group::build(group, uuid, last, front) } -/// The plan for a `find` whose query is `front`, a collection scan of the -/// collection whose UUID is `uuid`. `None` when its filter is outside what -/// is built here. -pub(crate) fn find_plan(uuid: &str, front: &Front) -> Option { - find::build(uuid, front) +/// The plan for a `find` whose query is `front` and whose projection is +/// `projection`, a collection scan of the collection whose UUID is `uuid`. +/// `None` when its filter or projection is outside what is built here. +pub(crate) fn find_plan(uuid: &str, front: &Front, projection: Option<&Document>) -> Option { + find::build(uuid, front, projection) } /// The plan mongod builds over a collection that does not exist. diff --git a/crates/secantus-commands/src/explain/sbe/print.rs b/crates/secantus-commands/src/explain/sbe/print.rs index 53e9c1f83..5202fe429 100644 --- a/crates/secantus-commands/src/explain/sbe/print.rs +++ b/crates/secantus-commands/src/explain/sbe/print.rs @@ -167,7 +167,64 @@ fn expr(e: &Expr, out: &mut Out) { expr(e, out); out.push(", int32)"); } + Expr::InList(s, _) => out.push(&format!("s{s}")), + Expr::ObjSpec { fields, open } => { + let (shape, ret) = if *open { + ("Open", "RetInput") + } else { + ("Closed", "RetNothing") + }; + out.push(&format!( + "MakeObjSpec([{}], {shape}, {ret}, 0)", + fields.join(", ") + )); + } + } +} + +/// An `InList` as the plan's environment prints it: `InList(1 "b" 2.5)`, +/// its values as the command gave them. `None` for a value this printer +/// is not known to write as mongod does. +pub(super) fn in_list(values: &[Bson]) -> Option { + let mut parts = Vec::with_capacity(values.len()); + for v in values { + parts.push(list_value(v)?); } + Some(format!("InList({})", parts.join(" "))) +} + +/// One `InList` value (measured 8.2.11: `2.0 1.5 3 -0.0 1e+20 "a b"`, +/// an int64 with no suffix, a double with a point or an exponent). +pub(super) fn list_value(v: &Bson) -> Option { + Some(match v { + Bson::Boolean(b) => b.to_string(), + Bson::Int32(n) => n.to_string(), + Bson::Int64(n) => n.to_string(), + Bson::Double(d) if d.is_finite() => { + let text = g17(*d); + // Written alike at any precision: six significant digits or + // fewer (what was measured). + let mantissa = text.split('e').next().unwrap_or(&text); + let digits = mantissa + .trim_start_matches('-') + .replace('.', "") + .trim_start_matches('0') + .trim_end_matches('0') + .len(); + if digits > 6 { + return None; + } + if text.contains(['.', 'e']) { + text + } else { + format!("{text}.0") + } + } + Bson::String(s) if !s.contains(['"', '\\']) && !s.chars().any(char::is_control) => { + format!("\"{s}\"") + } + _ => return None, + }) } fn op_text(op: BinOp) -> &'static str { diff --git a/crates/secantus-commands/src/ftdc.rs b/crates/secantus-commands/src/ftdc.rs index 5ccb2ad65..b444ad473 100644 --- a/crates/secantus-commands/src/ftdc.rs +++ b/crates/secantus-commands/src/ftdc.rs @@ -25,8 +25,10 @@ //! `diagnosticDataCollectionEnabled` and `diagnosticDataCollectionPeriodMillis` //! switch the sampling and set its period, as on mongod: a new period takes //! effect at its next boundary, and while sampling is off `getDiagnosticData` -//! keeps answering the last sample. The samples are not written to -//! `diagnostic.data` (RM-417). +//! keeps answering the last sample. Each sample is also written to +//! `/diagnostic.data` in mongod's file format (`crate::ftdc_file`, +//! RM-417), every file opening with the metadata collectors: `buildInfo`, +//! `getCmdLineOpts` and `hostInfo`. use std::sync::Mutex; use std::time::Duration; @@ -57,6 +59,11 @@ pub fn schedule(params: &crate::params::ServerParams) -> (bool, Duration) { #[derive(Default)] pub struct Ftdc { latest: Mutex, + /// The archive writer, opened at the first sample of a server with a + /// dbpath; `None` again after a write fails. + writer: Mutex>, + /// Whether the writer has been opened (or found impossible) yet. + opened: std::sync::atomic::AtomicBool, } impl Ftdc { @@ -75,8 +82,58 @@ impl Ftdc { /// Take a sample through `ctx` and keep it as the latest. pub fn collect(&self, ctx: &mut CommandContext) { let sample = sample(ctx); + self.write(&sample, ctx); *self.latest.lock().unwrap_or_else(|e| e.into_inner()) = sample; } + + /// Write `sample` to `diagnostic.data`. A write that fails is logged and + /// stops the archive, as mongod's FTDC stops writing after an error; + /// the in-memory sample carries on. + fn write(&self, sample: &Document, ctx: &mut CommandContext) { + use std::sync::atomic::Ordering; + let mut writer = self.writer.lock().unwrap_or_else(|e| e.into_inner()); + if !self.opened.swap(true, Ordering::SeqCst) { + let dir = ctx.storage().ok().and_then(|s| s.data_path()); + *writer = + dir.map(|d| crate::ftdc_file::Writer::new(crate::ftdc_file::directory_of(&d))); + } + let Some(w) = writer.as_mut() else { return }; + let limits = crate::ftdc_file::Limits::from_params(ctx.server_params.as_deref()); + if let Err(e) = w.write(sample, limits, || metadata(ctx)) { + log::error!( + "FTDC: writing diagnostic.data failed, no further samples are written: {e}" + ); + *writer = None; + } + } + + /// Write the chunk in progress, when the server stops. + pub fn close(&self) { + let mut writer = self.writer.lock().unwrap_or_else(|e| e.into_inner()); + if let Some(w) = writer.as_mut() { + if let Err(e) = w.close() { + log::error!("FTDC: closing diagnostic.data failed: {e}"); + } + } + *writer = None; + } +} + +/// The metadata at the head of every archive file (mongod's +/// `registerStartupCollectors`): `buildInfo`, `getCmdLineOpts`, `hostInfo`. +fn metadata(ctx: &mut CommandContext) -> Document { + let mut out = doc! {"start": DateTime::now()}; + let start = DateTime::now(); + let build = crate::handshake::build_info(&doc! {"buildInfo": 1}, ctx); + out.insert("buildInfo", stamped(start, build)); + let start = DateTime::now(); + let opts = crate::diagnostics::get_cmd_line_opts(&doc! {"getCmdLineOpts": 1}, ctx); + out.insert("getCmdLineOpts", stamped(start, opts)); + let start = DateTime::now(); + let host = crate::diagnostics::host_info(&doc! {"hostInfo": 1}, ctx); + out.insert("hostInfo", stamped(start, host)); + out.insert("end", DateTime::now()); + out } /// The next sampling time after `now_ms` (milliseconds since the epoch): diff --git a/crates/secantus-commands/src/ftdc_file.rs b/crates/secantus-commands/src/ftdc_file.rs new file mode 100644 index 000000000..b712616d7 --- /dev/null +++ b/crates/secantus-commands/src/ftdc_file.rs @@ -0,0 +1,546 @@ +//! mongod's FTDC archive: the samples written to `/diagnostic.data` +//! (RM-417), in mongod's own file format, so its decoders read them. +//! +//! A file is a run of BSON documents, each `{_id: , type: , ...}` +//! (measured on 8.2.11, 2026-11-14): +//! +//! - `type: 0`, metadata, `doc: {start, buildInfo, getCmdLineOpts, +//! hostInfo, end}`: the first document of every file; +//! - `type: 1`, a metric chunk, `data: BinData(0)`: a little-endian `u32` +//! of the uncompressed length, then the zlib of the chunk's first sample +//! (the reference document), a `u32` count of the metrics extracted from +//! it, a `u32` count of the samples after it, and the samples as deltas, +//! metric by metric, each an unsigned LEB128 varint, a run of zero deltas +//! written as `0` and the run's length less one (`FTDCCompressor`). +//! +//! A chunk ends when it holds `diagnosticDataCollectionSamplesPerChunk` +//! samples or a sample's shape differs from its reference document. Every +//! `diagnosticDataCollectionSamplesPerInterimUpdate` samples the chunk so far +//! is rewritten to `metrics.interim`, which the next start copies into its +//! new file. A file past `diagnosticDataCollectionFileSizeMB` is closed and +//! the next sample opens `metrics.-`; the oldest files are +//! removed while the directory is past `diagnosticDataCollectionDirectorySizeMB`. + +use std::fs::{self, File, OpenOptions}; +use std::io::{self, Write}; +use std::path::{Path, PathBuf}; + +use bson::{doc, spec::BinarySubtype, Binary, Bson, DateTime, Document}; + +/// The interim file's name. +pub const INTERIM: &str = "metrics.interim"; + +/// The writer's limits, from the server's parameters. +#[derive(Clone, Copy, Debug)] +pub struct Limits { + pub samples_per_chunk: usize, + pub samples_per_interim: usize, + pub file_bytes: u64, + pub directory_bytes: u64, +} + +impl Limits { + /// The limits `params` set now, mongod's defaults without them. + pub fn from_params(params: Option<&crate::params::ServerParams>) -> Self { + let n = + |name, default: i64| params.map_or(default, |p| p.number(name, default)).max(1) as u64; + Self { + samples_per_chunk: n(SAMPLES_PER_CHUNK, 300) as usize, + samples_per_interim: n(SAMPLES_PER_INTERIM, 10) as usize, + file_bytes: n(FILE_SIZE_MB, 10) * 1024 * 1024, + directory_bytes: n(DIRECTORY_SIZE_MB, 250) * 1024 * 1024, + } + } +} + +pub const DIRECTORY_PATH: &str = "diagnosticDataCollectionDirectoryPath"; +pub const DIRECTORY_SIZE_MB: &str = "diagnosticDataCollectionDirectorySizeMB"; +pub const FILE_SIZE_MB: &str = "diagnosticDataCollectionFileSizeMB"; +pub const SAMPLES_PER_CHUNK: &str = "diagnosticDataCollectionSamplesPerChunk"; +pub const SAMPLES_PER_INTERIM: &str = "diagnosticDataCollectionSamplesPerInterimUpdate"; + +/// The `diagnostic.data` directory of a store at `dbpath`. +pub fn directory_of(dbpath: &str) -> String { + Path::new(dbpath) + .join("diagnostic.data") + .to_string_lossy() + .into_owned() +} + +/// Append `element`'s metrics to `out` (mongod's +/// `extractMetricsFromDocument`): a number as a 64-bit integer, a bool as +/// 0 or 1, a date as its milliseconds, a timestamp as its seconds and then +/// its increment, a document or array by its own elements. +fn metrics_of(value: &Bson, out: &mut Vec) { + match value { + Bson::Double(d) => out.push(*d as i64), + Bson::Int32(n) => out.push(i64::from(*n)), + Bson::Int64(n) => out.push(*n), + Bson::Decimal128(d) => out.push(d.to_string().parse::().unwrap_or(0.0) as i64), + Bson::Boolean(b) => out.push(i64::from(*b)), + Bson::DateTime(t) => out.push(t.timestamp_millis()), + Bson::Timestamp(t) => { + out.push(i64::from(t.time)); + out.push(i64::from(t.increment)); + } + Bson::Document(d) => d.values().for_each(|v| metrics_of(v, out)), + Bson::Array(a) => a.iter().for_each(|v| metrics_of(v, out)), + _ => {} + } +} + +fn is_metric(value: &Bson) -> bool { + matches!( + value, + Bson::Double(_) + | Bson::Int32(_) + | Bson::Int64(_) + | Bson::Decimal128(_) + | Bson::Boolean(_) + | Bson::DateTime(_) + | Bson::Timestamp(_) + ) +} + +/// Whether `sample` has `reference`'s shape: the same names in the same +/// order, metrics where it has metrics, and nested documents and arrays of +/// the same shape. A sample that does not starts a new chunk. +fn same_shape(reference: &Bson, sample: &Bson) -> bool { + match (reference, sample) { + (Bson::Document(r), Bson::Document(s)) => { + r.len() == s.len() + && r.iter() + .zip(s.iter()) + .all(|((rk, rv), (sk, sv))| rk == sk && same_shape(rv, sv)) + } + (Bson::Array(r), Bson::Array(s)) => { + r.len() == s.len() && r.iter().zip(s).all(|(a, b)| same_shape(a, b)) + } + (Bson::Document(_) | Bson::Array(_), _) | (_, Bson::Document(_) | Bson::Array(_)) => false, + (r, s) => { + is_metric(r) == is_metric(s) + && matches!(r, Bson::Timestamp(_)) == matches!(s, Bson::Timestamp(_)) + } + } +} + +fn varint(mut v: u64, out: &mut Vec) { + loop { + let byte = (v & 0x7f) as u8; + v >>= 7; + if v == 0 { + out.push(byte); + return; + } + out.push(byte | 0x80); + } +} + +/// The samples of one chunk: its reference document and the metrics of +/// every sample after it. +#[derive(Default)] +pub struct Chunk { + reference: Option, + reference_metrics: Vec, + samples: Vec>, +} + +impl Chunk { + /// How many samples the chunk holds, its reference included. + pub fn len(&self) -> usize { + self.reference + .as_ref() + .map_or(0, |_| 1 + self.samples.len()) + } + + pub fn is_empty(&self) -> bool { + self.reference.is_none() + } + + /// Add `sample` if it has the reference's shape (or the chunk is + /// empty); `false`, adding nothing, if it does not. + pub fn add(&mut self, sample: &Document) -> bool { + let mut metrics = Vec::new(); + metrics_of(&Bson::Document(sample.clone()), &mut metrics); + match &self.reference { + None => { + self.reference = Some(sample.clone()); + self.reference_metrics = metrics; + true + } + Some(r) => { + let fits = metrics.len() == self.reference_metrics.len() + && same_shape(&Bson::Document(r.clone()), &Bson::Document(sample.clone())); + if fits { + self.samples.push(metrics); + } + fits + } + } + } + + /// The chunk as mongod's `type: 1` document. + pub fn to_document(&self) -> io::Result { + let reference = self + .reference + .as_ref() + .expect("an empty chunk is never written"); + let mut raw = secantus_core::bsondoc::to_vec(reference).map_err(io::Error::other)?; + raw.extend((self.reference_metrics.len() as u32).to_le_bytes()); + raw.extend((self.samples.len() as u32).to_le_bytes()); + let mut zeros = 0u64; + for (i, first) in self.reference_metrics.iter().enumerate() { + let mut previous = *first; + for sample in &self.samples { + let delta = sample[i].wrapping_sub(previous) as u64; + previous = sample[i]; + if delta == 0 { + zeros += 1; + continue; + } + if zeros > 0 { + varint(0, &mut raw); + varint(zeros - 1, &mut raw); + zeros = 0; + } + varint(delta, &mut raw); + } + } + if zeros > 0 { + varint(0, &mut raw); + varint(zeros - 1, &mut raw); + } + let mut data = (raw.len() as u32).to_le_bytes().to_vec(); + let mut z = flate2::write::ZlibEncoder::new(&mut data, flate2::Compression::default()); + z.write_all(&raw)?; + z.finish()?; + let id = match reference.get("start") { + Some(Bson::DateTime(t)) => *t, + _ => DateTime::now(), + }; + Ok(doc! { + "_id": id, + "type": 1_i32, + "data": Binary { subtype: BinarySubtype::Generic, bytes: data }, + }) + } +} + +/// The open archive file. +struct Archive { + file: File, + size: u64, +} + +/// The writer of one `diagnostic.data` directory. +pub struct Writer { + dir: PathBuf, + archive: Option, + chunk: Chunk, + since_interim: usize, +} + +fn append(file: &mut File, document: &Document) -> io::Result { + let bytes = secantus_core::bsondoc::to_vec(document).map_err(io::Error::other)?; + file.write_all(&bytes)?; + Ok(bytes.len() as u64) +} + +impl Writer { + pub fn new(dir: impl Into) -> Self { + Self { + dir: dir.into(), + archive: None, + chunk: Chunk::default(), + since_interim: 0, + } + } + + /// The archive file's name for a file opened at `now`: mongod's + /// `metrics.-`, `n` counting past a name in use. + fn archive_name(&self, now: DateTime) -> PathBuf { + let stamp = now + .try_to_rfc3339_string() + .unwrap_or_default() + .get(..19) + .unwrap_or("") + .replace(':', "-"); + (0..) + .map(|n| self.dir.join(format!("metrics.{stamp}Z-{n:05}"))) + .find(|p| !p.exists()) + .expect("an unused name") + } + + /// Open a new archive file, its metadata first and then the chunk a + /// previous run left in `metrics.interim`. + fn open(&mut self, metadata: &Document) -> io::Result<()> { + fs::create_dir_all(&self.dir)?; + let now = DateTime::now(); + let path = self.archive_name(now); + let mut file = OpenOptions::new() + .create_new(true) + .append(true) + .open(path)?; + let mut size = append( + &mut file, + &doc! {"_id": now, "type": 0_i32, "doc": metadata.clone()}, + )?; + let interim = self.dir.join(INTERIM); + if self.chunk.is_empty() && interim.exists() { + let bytes = fs::read(&interim)?; + file.write_all(&bytes)?; + size += bytes.len() as u64; + fs::remove_file(&interim)?; + } + file.sync_data()?; + self.archive = Some(Archive { file, size }); + Ok(()) + } + + /// Write the chunk to the archive and start the next one. + fn flush(&mut self) -> io::Result<()> { + if self.chunk.is_empty() { + return Ok(()); + } + let document = self.chunk.to_document()?; + let archive = self.archive.as_mut().expect("opened before a flush"); + archive.size += append(&mut archive.file, &document)?; + archive.file.sync_data()?; + self.chunk = Chunk::default(); + self.since_interim = 0; + let interim = self.dir.join(INTERIM); + if interim.exists() { + fs::remove_file(interim)?; + } + Ok(()) + } + + /// Write `sample`; `metadata` is asked for when a file is opened. + pub fn write( + &mut self, + sample: &Document, + limits: Limits, + metadata: impl FnOnce() -> Document, + ) -> io::Result<()> { + if self.archive.is_none() { + self.open(&metadata())?; + } + if !self.chunk.add(sample) { + self.flush()?; + self.chunk.add(sample); + } + if self.chunk.len() >= limits.samples_per_chunk.max(2) { + self.flush()?; + } else { + self.since_interim += 1; + if self.since_interim >= limits.samples_per_interim { + self.since_interim = 0; + let document = self.chunk.to_document()?; + let tmp = self.dir.join("metrics.interim.temp"); + fs::write( + &tmp, + secantus_core::bsondoc::to_vec(&document).map_err(io::Error::other)?, + )?; + fs::rename(tmp, self.dir.join(INTERIM))?; + } + } + if self + .archive + .as_ref() + .is_some_and(|a| a.size >= limits.file_bytes) + { + self.archive = None; + self.prune(limits.directory_bytes)?; + } + Ok(()) + } + + /// Remove the oldest archive files while the directory is past `limit`. + fn prune(&self, limit: u64) -> io::Result<()> { + let mut files: Vec<(String, u64)> = fs::read_dir(&self.dir)? + .filter_map(Result::ok) + .filter_map(|e| { + let name = e.file_name().to_string_lossy().into_owned(); + let archive = name.starts_with("metrics.") && !name.starts_with(INTERIM); + archive.then(|| (name, e.metadata().map(|m| m.len()).unwrap_or(0))) + }) + .collect(); + files.sort(); + let mut total: u64 = files.iter().map(|f| f.1).sum(); + for (name, size) in files { + if total <= limit { + break; + } + fs::remove_file(self.dir.join(name))?; + total -= size; + } + Ok(()) + } + + /// Write the chunk in progress, as mongod does when it stops. + pub fn close(&mut self) -> io::Result<()> { + if self.archive.is_some() { + self.flush()?; + } + self.archive = None; + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::Read; + + fn limits(chunk: usize, interim: usize) -> Limits { + Limits { + samples_per_chunk: chunk, + samples_per_interim: interim, + file_bytes: 10 << 20, + directory_bytes: 250 << 20, + } + } + + fn decode_chunk(d: &Document) -> (Document, u32, u32, Vec) { + let bytes = &d.get_binary_generic("data").unwrap()[..]; + let length = u32::from_le_bytes(bytes[..4].try_into().unwrap()) as usize; + let mut raw = Vec::new(); + flate2::read::ZlibDecoder::new(&bytes[4..]) + .read_to_end(&mut raw) + .unwrap(); + assert_eq!(raw.len(), length); + let reference = secantus_core::bsondoc::from_bytes(&raw).unwrap(); + let at = secantus_core::bsondoc::to_vec(&reference).unwrap().len(); + let n = |i: usize| u32::from_le_bytes(raw[i..i + 4].try_into().unwrap()); + (reference, n(at), n(at + 4), raw[at + 8..].to_vec()) + } + + fn documents(path: &Path) -> Vec { + let bytes = fs::read(path).unwrap(); + let mut out = Vec::new(); + let mut at = 0; + while at < bytes.len() { + let d = secantus_core::bsondoc::from_bytes(&bytes[at..]).unwrap(); + at += secantus_core::bsondoc::to_vec(&d).unwrap().len(); + out.push(d); + } + out + } + + #[test] + fn deltas_are_varints_by_metric_with_zero_runs() { + // mongod 8.2.11 wrote `dc07 ee07 e007 00 02 01 01 01` for a `start` + // advancing 988, 1006 and 992 ms, three unchanged metrics, then one + // rising by 1 each sample. + let mut chunk = Chunk::default(); + let mut t = 1_000_000; + for (i, step) in [0, 988, 1006, 992].into_iter().enumerate() { + t += step; + chunk.add(&doc! { + "start": DateTime::from_millis(t), + "a": 5_i32, "b": true, "c": 7_i64, "d": i as i64, + }); + } + let (reference, metrics, samples, deltas) = decode_chunk(&chunk.to_document().unwrap()); + assert_eq!(reference.get_i32("a").unwrap(), 5); + assert_eq!((metrics, samples), (5, 3)); + let mut want = Vec::new(); + for v in [988, 1006, 992, 0, 8, 1, 1, 1] { + varint(v, &mut want); + } + assert_eq!(deltas, want); + } + + #[test] + fn a_sample_of_another_shape_starts_a_chunk() { + let mut chunk = Chunk::default(); + assert!(chunk.add(&doc! {"a": 1_i32, "s": "x"})); + assert!(chunk.add(&doc! {"a": 2_i64, "s": "y"})); + assert!(!chunk.add(&doc! {"b": 2_i32, "s": "y"})); + assert!(!chunk.add(&doc! {"a": "no", "s": "y"})); + assert!(!chunk.add(&doc! {"a": {"t": bson::Timestamp {time: 1, increment: 2}}, "s": "y"})); + } + + #[test] + fn a_file_starts_with_metadata_and_holds_full_chunks() { + let dir = tempfile::tempdir().unwrap(); + let mut w = Writer::new(dir.path()); + for i in 0..5_i32 { + w.write(&doc! {"n": i}, limits(2, 10), || doc! {"buildInfo": {}}) + .unwrap(); + } + let names: Vec = fs::read_dir(dir.path()) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + assert_eq!(names.len(), 1, "{names:?}"); + let docs = documents(&dir.path().join(&names[0])); + assert_eq!( + docs.iter() + .map(|d| d.get_i32("type").unwrap()) + .collect::>(), + [0, 1, 1] + ); + w.close().unwrap(); + let docs = documents(&dir.path().join(&names[0])); + assert_eq!(docs.len(), 4); + assert_eq!(decode_chunk(&docs[3]).0, doc! {"n": 4}); + } + + #[test] + fn the_interim_file_is_carried_into_the_next_run() { + let dir = tempfile::tempdir().unwrap(); + let mut w = Writer::new(dir.path()); + for i in 0..3_i32 { + w.write(&doc! {"n": i}, limits(300, 2), Document::new) + .unwrap(); + } + assert!(dir.path().join(INTERIM).exists()); + drop(w); // a crash: nothing flushed + let mut next = Writer::new(dir.path()); + next.write(&doc! {"n": 9}, limits(300, 2), Document::new) + .unwrap(); + assert!(!dir.path().join(INTERIM).exists()); + let carried: Vec = fs::read_dir(dir.path()) + .unwrap() + .flat_map(|e| documents(&e.unwrap().path())) + .filter(|d| d.get_i32("type") == Ok(1)) + .collect(); + assert_eq!(carried.len(), 1); + assert_eq!(decode_chunk(&carried[0]).2, 1, "the interim's two samples"); + } + + #[test] + fn a_full_file_rotates_and_the_directory_is_pruned() { + let dir = tempfile::tempdir().unwrap(); + let mut w = Writer::new(dir.path()); + let small = Limits { + samples_per_chunk: 2, + samples_per_interim: 100, + file_bytes: 1, + directory_bytes: 1, + }; + for i in 0..6_i32 { + w.write(&doc! {"n": i}, small, Document::new).unwrap(); + } + let left = fs::read_dir(dir.path()).unwrap().count(); + assert_eq!(left, 0, "every closed file is past the directory's limit"); + } + + #[test] + fn archive_names_are_utc_seconds_and_count_past_a_clash() { + let dir = tempfile::tempdir().unwrap(); + let w = Writer::new(dir.path()); + let t = DateTime::from_millis(1_794_121_279_040); + let first = w.archive_name(t); + assert_eq!( + first.file_name().unwrap(), + "metrics.2026-11-08T07-01-19Z-00000" + ); + fs::write(&first, b"").unwrap(); + assert_eq!( + w.archive_name(t).file_name().unwrap(), + "metrics.2026-11-08T07-01-19Z-00001" + ); + } +} diff --git a/crates/secantus-commands/src/lib.rs b/crates/secantus-commands/src/lib.rs index d521bc1fa..955bf467c 100644 --- a/crates/secantus-commands/src/lib.rs +++ b/crates/secantus-commands/src/lib.rs @@ -74,6 +74,7 @@ pub mod failpoints; pub mod find; pub mod findandmodify; pub mod ftdc; +pub mod ftdc_file; pub mod handshake; mod hostinfo; pub use hostinfo::mem_size_mb; diff --git a/crates/secantus-commands/src/params.rs b/crates/secantus-commands/src/params.rs index 1b70d9368..fdc45188a 100644 --- a/crates/secantus-commands/src/params.rs +++ b/crates/secantus-commands/src/params.rs @@ -134,6 +134,26 @@ const SETTABLE: &[(&str, ParamKind)] = &[ "diagnosticDataCollectionPeriodMillis", ParamKind::Int32AtLeast(100), ), + // The FTDC archive's limits (`crate::ftdc_file`, RM-417), read at every + // sample, with mongod 8.2.11's minimums. Its directory is fixed once + // set: a change is 220 (`ftdc_path_refusal`). + ("diagnosticDataCollectionDirectoryPath", ParamKind::Bool), + ( + "diagnosticDataCollectionDirectorySizeMB", + ParamKind::Int32AtLeast(10), + ), + ( + "diagnosticDataCollectionFileSizeMB", + ParamKind::Int32AtLeast(1), + ), + ( + "diagnosticDataCollectionSamplesPerChunk", + ParamKind::Int32AtLeast(2), + ), + ( + "diagnosticDataCollectionSamplesPerInterimUpdate", + ParamKind::Int32AtLeast(2), + ), // Refuse a query the planner would answer with a collection scan // (`crate::notablescan`); read on every query, so it is also kept in an // atomic beside the stored value. @@ -777,16 +797,17 @@ pub(crate) fn integral(value: &Bson, int32: bool) -> Result { if !n.is_finite() { return Err("Unable to coerce NaN/Inf to integral type"); } - let n = n.trunc(); - let (low, high) = if int32 { - (f64::from(i32::MIN), f64::from(i32::MAX)) + // mongod bounds the double before truncating it (RM-541, measured + // 8.2.11): `2147483647.9` and `-2147483648.5` are out of an int's range. + let out_of_bounds = if int32 { + n < f64::from(i32::MIN) || n > f64::from(i32::MAX) } else { - (i64::MIN as f64, i64::MAX as f64) + n < i64::MIN as f64 || n >= i64::MAX as f64 }; - if n < low || n >= high + 1.0 { + if out_of_bounds { return Err("Out of bounds coercing to integral value"); } - Ok(n as i64) + Ok(n.trunc() as i64) } /// mongod's words for a value `coerce` refused (measured 8.2.11). @@ -922,6 +943,40 @@ fn coerce_named( coerce(kind, value).ok_or_else(|| refusal(kind, name, value)) } +/// mongod's refusal to move the FTDC directory once it is set, which it is +/// from the start on a server with a dbpath (8.2.11, 2026-11-14: any value, +/// of any type). +fn ftdc_path_refusal(ctx: &CommandContext) -> CommandError { + let path = crate::serverinfo::ftdc_directory(ctx); + CommandError::new( + 220, + "FTDCPathAlreadySet", + format!("FTDC path has already been set to '{path}'. It cannot be changed."), + ) +} + +/// mongod's check that an FTDC file fits in its directory, made AFTER the +/// new value is stored: on 8.2.11 (2026-11-14) a refused +/// `diagnosticDataCollectionFileSizeMB: 300` still reads back as 300, and +/// the next change reports it as `was`. +fn ftdc_sizes_agree(params: &ServerParams, name: &str) -> Result<(), CommandError> { + use crate::ftdc_file::{DIRECTORY_SIZE_MB, FILE_SIZE_MB}; + let file = params.number(FILE_SIZE_MB, 10); + let directory = params.number(DIRECTORY_SIZE_MB, 250); + let why = match name { + FILE_SIZE_MB if file > directory => format!( + "{FILE_SIZE_MB} must be less than or equal to '{directory}' which is the current \ + value of {DIRECTORY_SIZE_MB}." + ), + DIRECTORY_SIZE_MB if directory < file => format!( + "{DIRECTORY_SIZE_MB} must be greater than or equal to '{file}' which is the \ + current value of {FILE_SIZE_MB}." + ), + _ => return Ok(()), + }; + Err(CommandError::new(2, "BadValue", why)) +} + /// `setParameter` — change a runtime-settable server parameter. pub fn set_parameter(doc: &Document, ctx: &mut CommandContext) -> HandlerResult { if ctx.db_name != "admin" { @@ -949,6 +1004,9 @@ pub fn set_parameter(doc: &Document, ctx: &mut CommandContext) -> HandlerResult // mongod validates every named parameter BEFORE applying any of them, so a // batch that names one bad parameter changes nothing. for (name, value) in &requested { + if name.as_str() == crate::ftdc_file::DIRECTORY_PATH { + return Err(ftdc_path_refusal(ctx)); + } // A parameter this server reports and does not let a client change. // Answering 20 and not 72 matters: 72 claims the parameter does not // exist, and `getParameter` would immediately contradict that. (mongod @@ -1053,6 +1111,7 @@ pub fn set_parameter(doc: &Document, ctx: &mut CommandContext) -> HandlerResult if let Some(s) = store.as_ref() { s.set(name, coerced.clone()); s.apply_execution(name, &coerced)?; + ftdc_sizes_agree(s, name)?; } } if Some(name.as_str()) == first { @@ -1068,6 +1127,20 @@ mod tests { use super::*; use bson::doc; + #[test] + fn an_int_bounds_the_double_before_truncating() { + // RM-541: mongod 8.2.11 takes 2147483647.0, refuses 2147483647.9. + assert_eq!(integral(&Bson::Double(2147483647.0), true), Ok(2147483647)); + assert_eq!( + integral(&Bson::Double(-2147483648.0), true), + Ok(-2147483648) + ); + let oob = Err("Out of bounds coercing to integral value"); + assert_eq!(integral(&Bson::Double(2147483647.9), true), oob); + assert_eq!(integral(&Bson::Double(-2147483648.5), true), oob); + assert_eq!(integral(&Bson::Double(-1.9), true), Ok(-1)); + } + fn ctx() -> CommandContext { let mut c = CommandContext::new(1); c.db_name = "admin".to_string(); diff --git a/crates/secantus-commands/src/query_settings.rs b/crates/secantus-commands/src/query_settings.rs index d59aaff42..3c4842d2b 100644 --- a/crates/secantus-commands/src/query_settings.rs +++ b/crates/secantus-commands/src/query_settings.rs @@ -27,8 +27,7 @@ //! and under `"sbe"` (`trySbeEngine`) a `find` SBE compiles runs there. //! //! Not yet: under `"sbe"` an `aggregate` and a `distinct` are pushed down -//! as without the setting, `showDebugQueryShape` gives no -//! `debugQueryShape`, and a write made to `config.clusterParameters` +//! as without the setting, and a write made to `config.clusterParameters` //! directly is read only by the next server start (RM-418). use std::cell::RefCell; @@ -710,14 +709,32 @@ pub(crate) fn validate_stage(spec: Option<&Bson>) -> Result Ok(show) } -/// `$querySettings`'s documents: every setting. `showDebugQueryShape` is -/// parsed but its `debugQueryShape` is not given (RM-418). +/// `$querySettings`'s documents: every setting. Under +/// `showDebugQueryShape` each one with a `representativeQuery` also +/// carries its `debugQueryShape`: the query's shape with each literal a +/// debug type string (`document_source_query_settings.cpp`). pub(crate) fn stage_rows( storage: &dyn crate::storage::Storage, spec: Option<&Bson>, ) -> Result, CommandError> { - validate_stage(spec)?; - Ok(entries_of(stored(storage)?.as_ref())) + let show = validate_stage(spec)?; + let mut rows = entries_of(stored(storage)?.as_ref()); + if show { + for row in &mut rows { + if let Some(shape) = debug_query_shape(row) { + row.insert("debugQueryShape", shape); + } + } + } + Ok(rows) +} + +/// The debug shape of a setting's `representativeQuery`, run against its +/// `$db`. +fn debug_query_shape(entry: &Document) -> Option { + let query = entry.get_document("representativeQuery").ok()?; + let db = query.get_str("$db").ok()?; + crate::query_shape::debug_shape(db, query) } // ------------------------------------------- the settings a read runs under -- diff --git a/crates/secantus-commands/src/serverinfo.rs b/crates/secantus-commands/src/serverinfo.rs index b1f2c8ae5..391c49506 100644 --- a/crates/secantus-commands/src/serverinfo.rs +++ b/crates/secantus-commands/src/serverinfo.rs @@ -674,6 +674,29 @@ fn listed(test_commands: bool) -> Vec<(&'static str, bool, Bson)> { true, Bson::Int32(1000), ), + // The FTDC archive (`crate::ftdc_file`, RM-417); the directory is + // the dbpath's, answered by `ftdc_directory`. + ( + "diagnosticDataCollectionDirectoryPath", + true, + Bson::String(String::new()), + ), + ( + "diagnosticDataCollectionDirectorySizeMB", + true, + Bson::Int32(250), + ), + ("diagnosticDataCollectionFileSizeMB", true, Bson::Int32(10)), + ( + "diagnosticDataCollectionSamplesPerChunk", + true, + Bson::Int32(300), + ), + ( + "diagnosticDataCollectionSamplesPerInterimUpdate", + true, + Bson::Int32(10), + ), // Whether the logical session cache refreshes and reaps at all. ( "disableLogicalSessionCacheRefresh", @@ -830,6 +853,16 @@ fn listed(test_commands: bool) -> Vec<(&'static str, bool, Bson)> { /// switch the server was started with until a client sets them, as mongod /// reports a `--setParameter` startup value. `None` for any other parameter, /// or a backend with no monitor. +/// The FTDC archive's directory, `/diagnostic.data` as the dbpath +/// was given; empty for a server with no dbpath. +pub(crate) fn ftdc_directory(ctx: &CommandContext) -> String { + ctx.storage() + .ok() + .and_then(|s| s.data_path()) + .map(|d| crate::ftdc_file::directory_of(&d)) + .unwrap_or_default() +} + pub(crate) fn ttl_monitor_param(ctx: &CommandContext, name: &str) -> Option { let (secs, enabled, _, _) = ctx.storage().ok()?.ttl_monitor()?; match name { @@ -921,6 +954,8 @@ pub fn get_parameter(doc: &Document, ctx: &mut CommandContext) -> HandlerResult // it recorded one. let value = if name == "featureCompatibilityVersion" { Bson::Document(doc! {"version": crate::cluster_admin::current_fcv(ctx)}) + } else if name == crate::ftdc_file::DIRECTORY_PATH { + Bson::String(ftdc_directory(ctx)) } else { value }; diff --git a/crates/secantus-server-py/Cargo.lock b/crates/secantus-server-py/Cargo.lock index 8fd5d8c9d..f381c2e68 100644 --- a/crates/secantus-server-py/Cargo.lock +++ b/crates/secantus-server-py/Cargo.lock @@ -2279,6 +2279,7 @@ version = "0.5.3-beta.196" dependencies = [ "base64 0.22.1", "bson", + "flate2", "hmac 0.12.1", "libc", "log", diff --git a/crates/secantus-server/src/lib.rs b/crates/secantus-server/src/lib.rs index 69c901c27..d19e1a6dd 100644 --- a/crates/secantus-server/src/lib.rs +++ b/crates/secantus-server/src/lib.rs @@ -618,6 +618,8 @@ fn ftdc_loop(shared: Arc) { thread::sleep(Duration::from_millis((next - now) as u64).min(FTDC_POLL)); } } + // mongod writes the chunk in progress to `diagnostic.data` as it stops. + shared.ftdc.close(); } fn accept_loop(listener: TcpListener, shared: Arc) { diff --git a/crates/secantus-storage-adapter/Cargo.lock b/crates/secantus-storage-adapter/Cargo.lock index ea8748e90..d516571bd 100644 --- a/crates/secantus-storage-adapter/Cargo.lock +++ b/crates/secantus-storage-adapter/Cargo.lock @@ -1071,6 +1071,7 @@ version = "0.5.3-beta.196" dependencies = [ "base64", "bson", + "flate2", "hmac", "libc", "log", diff --git a/crates/secantusdb/Cargo.lock b/crates/secantusdb/Cargo.lock index 0496d2406..4dade7370 100644 --- a/crates/secantusdb/Cargo.lock +++ b/crates/secantusdb/Cargo.lock @@ -2255,6 +2255,7 @@ version = "0.5.3-beta.196" dependencies = [ "base64", "bson", + "flate2", "hmac 0.12.1", "libc", "log", diff --git a/tasks/backlog-rust-mongodb.md b/tasks/backlog-rust-mongodb.md index 219e30bda..af8acc9fa 100644 --- a/tasks/backlog-rust-mongodb.md +++ b/tasks/backlog-rust-mongodb.md @@ -2077,56 +2077,32 @@ live. Not filed here: `$rankFusion` / `$score` / `$scoreFusion` (RM-198), Carried over from RM-416, 2026-11-14. Verify: tests/backlog/test_rm_219_diag_query_settings.py::test_db_check_extra_index_keys_remainder, tests/backlog/test_rm_219_diag_query_settings.py::test_apply_ops_other_entries Backlog-ID: RM-688 -- [ ] **`applyOps` applies a `u` replacement of a missing document.** A `u` - entry whose `o` is a whole document with `o2`'s `_id`, for a document - that does not exist and without `b: true`, is inserted here; mongod - 8.2.11 does not apply it (`false` in `results`, the command 8), as this - server's own module comment says. A `u` upsert of a document past - 16 MB is 10334 "object to insert too large" here, 17420 "Document to - upsert is larger than 16777216" on mongod, and an `i` entry replacing - an existing document with one past 16 MB is 10334 here, 17419 on - mongod. Found 2026-11-10 while working RM-416. - Verify: tests/backlog/test_rm_219_diag_query_settings.py::test_apply_ops_update_entries - Backlog-ID: RM-540 -- [ ] **FTDC is a sample in memory only.** Since 2026-11-10 the sample - carries `transportLayerStats` and `diagnosticDataCollectionEnabled` / - `diagnosticDataCollectionPeriodMillis` switch it and set its period. - Open: nothing is written to `diagnostic.data` (mongod 8.2.11 keeps - `metrics.