diff --git a/CHANGELOG.md b/CHANGELOG.md index da3076c..25b454a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,40 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added +- **Parquet & Arrow interoperability** (palette → "Open Parquet/Arrow…" and + "Export as Parquet/Arrow…"): open and export typed columnar datasets — + Apache Parquet, Arrow IPC files (Feather v2 is the Arrow IPC file format), + and Arrow IPC streams — preserving types and nulls. Opening a `.parquet` / + `.arrow` / `.feather` / `.ipc` file (drag-and-drop, "Open file…", or the + command) first shows an inspect dialog: container format, row and + row-group/batch counts, compression codec, the schema mapped to the F31 + logical types (nested fields indented, timezones shown), and an + editable-memory estimate — before anything is loaded. Choose read-only + (an indexed columnar backing with windowed reads over row groups / record + batches and a bounded decoded-block cache, so the grid, filters, and + export work with bounded memory on multi-gigabyte files) or convert to + editable behind an explicit memory check. Signed and unsigned 64-bit + integers (so `u64::MAX` round-trips losslessly), exact decimal + precision+scale, floats, booleans, dates, timestamps with their time-zone + metadata, and UTF-8 strings all survive intact, and a NULL stays distinct + from an empty string end to end (editable opens preserve the distinction + through collision-free per-column null tokens). Structs flatten to stable + path-based column names; each list/map/struct field takes an explicit + per-field policy (keep as JSON, explode into rows on an editable open, or + drop). Equality and range filters on numeric/date columns of indexed + parquet documents skip whole row groups using their statistics, with + results identical to a full scan. Export any scope (all rows, the filtered + view, selected rows / columns / range) to Parquet (uncompressed, Snappy, + or Zstd, with a configurable row-group size), an Arrow IPC file, or an + Arrow IPC stream, as a cancellable job through the atomic-save pipeline; + typed export maps each column's declared logical type to the matching + arrow type (Int64/UInt64, Decimal128 with unified precision/scale, Date32, + microsecond or nanosecond timestamps carrying the schema's time zone), + while null tokens and columnar NULLs export as real nulls distinct from + empty strings. Cells that cannot be represented under the declared types + are written as NULL and counted into a per-column warning report; columns + without a schema export as text verbatim. A columnar document opens + unsaved so a later Save can never overwrite the binary source with CSV. - **Row bookmarks, tags & notes** (F40): mark and annotate records without touching the source data. Star or flag a row, apply multiple named tags (a per-document tag namespace with usage counts), and attach a row note or @@ -199,6 +233,16 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). the backend), advisory accepts it and records a bounded, retrievable issue — while schema edits themselves never touch the undo stack. +### Fixed + +- **Project open now restores Parquet / Arrow sources non-interactively**: + reopening a project that referenced a columnar source no longer routes it + through the interactive inspect dialog (which returned without creating a + tab, so the source was left missing until the user manually confirmed the + dialog while the project baseline had already advanced). A restore reopens a + columnar source directly as an indexed read-only document with default + policies, matching the non-interactive restore of every other source type. + ### Internal - **Shared tabular contracts**: new backend `TabularSource` / diff --git a/README.md b/README.md index 9534e28..f207867 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,15 @@ and faithful on large, real-world delimited files.** detects the shape, infers columns, counts missing vs explicit-null cells, and lets you flatten, preserve, join, or explode nested objects and arrays. JSON Lines opens read-only with bounded memory. +- **Open Parquet / Arrow** — inspect an Apache Parquet, Arrow IPC file + (Feather v2), or Arrow IPC stream before opening: format, row and + row-group/batch counts, compression codec, the schema mapped to logical + types (with nested fields and timezones), and the editable-memory + estimate. Open read-only (indexed, bounded memory) or convert to editable; + nested list/map/struct fields take a per-field policy (keep as JSON, + explode into rows, or drop). Signed/unsigned 64-bit integers, exact + decimal precision/scale, timestamps with timezone, and null-vs-empty-string + all survive intact. - Auto-detect the **delimiter** (comma, tab, semicolon, pipe) with a manual / custom override — plus an **advanced import** for preambles, comment lines, custom quoting/escaping, multi-row headers, and footers. @@ -106,6 +115,12 @@ and faithful on large, real-world delimited files.** array of arrays, or JSON Lines; typed columns emit real numbers and booleans, nested objects rebuild from dotted-path columns, and duplicate output paths are rejected before writing. +- **Export as Parquet / Arrow** — write any export scope to Apache Parquet + (uncompressed, Snappy, or Zstd, with a configurable row-group size), an + Arrow IPC file (Feather v2), or an Arrow IPC stream. Typed export maps each + column's declared logical type to the matching arrow type (preserving 64-bit + integer widths, decimal precision/scale, and timestamp timezones); cells + that can't be represented are written as NULL and reported per column. **Reliability** diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index 09ca102..c6c1a3a 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -8,6 +8,20 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "const-random", + "getrandom 0.3.4", + "once_cell", + "version_check", + "zerocopy", +] + [[package]] name = "aho-corasick" version = "1.1.4" @@ -77,6 +91,180 @@ dependencies = [ "x11rb", ] +[[package]] +name = "arrow" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b952ca5a8046ad741b60f142d6eca4aeebcad615694202bc64c5341f23e32c5b" +dependencies = [ + "arrow-arith", + "arrow-array", + "arrow-buffer", + "arrow-cast", + "arrow-data", + "arrow-ipc", + "arrow-ord", + "arrow-row", + "arrow-schema", + "arrow-select", + "arrow-string", +] + +[[package]] +name = "arrow-arith" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "64a13b8d3008c4e9063c597a08f46446fe3fd5789277127672d6c0bdbb43b1ff" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "chrono", + "num-traits", +] + +[[package]] +name = "arrow-array" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9486151b2f0785bafc6fa04fc5c99fcb4495455662e58787ea32eaaed33c4192" +dependencies = [ + "ahash", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "chrono", + "half", + "hashbrown 0.17.1", + "num-complex", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-buffer" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4776577a87794bfdf0b4e90e2ea12454fa7738ea2823c4be5b9d1851da7b434" +dependencies = [ + "bytes", + "half", + "num-bigint", + "num-traits", +] + +[[package]] +name = "arrow-cast" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9ad451ce4f98710828a455b96991b8f031deb2e67f5fcad6773f017e4a69c3a" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-ord", + "arrow-schema", + "arrow-select", + "atoi", + "base64 0.22.1", + "chrono", + "half", + "lexical-core", + "num-traits", + "ryu", +] + +[[package]] +name = "arrow-data" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b38fe43e2e8704360f1464e6e8cc4fc381ef02cc4fb0192afa8df1aaa0115c66" +dependencies = [ + "arrow-buffer", + "arrow-schema", + "half", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-ipc" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29dac499fcbc6ba74ee0324057821d381929a48526a3966bd9dffb44aa06d98c" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", + "flatbuffers", +] + +[[package]] +name = "arrow-ord" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e13dbdc2a9c053c10c7baa6e30faee04a180aa7ce88e471835850ce37abd20b" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", +] + +[[package]] +name = "arrow-row" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4d5a1f8c733d15260b305683472ee8ad89c62cbd706703ca873b90d051b41592" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "half", +] + +[[package]] +name = "arrow-schema" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9e4969dc350d571766247143ab36a5187d095d3d3690970408bc630d47c69e5" + +[[package]] +name = "arrow-select" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "402770dba90865359d98d1ef92ef16e23d75c0cca9c2c880c8a05468b7743bf9" +dependencies = [ + "ahash", + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "num-traits", +] + +[[package]] +name = "arrow-string" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2b0afbb8b9016700938291123df30838b89decc3213dba00852021988b170d3" +dependencies = [ + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-schema", + "arrow-select", + "memchr", + "num-traits", + "regex", + "regex-syntax", +] + [[package]] name = "async-broadcast" version = "0.7.2" @@ -231,6 +419,15 @@ dependencies = [ "system-deps", ] +[[package]] +name = "atoi" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f28d99ec8bfea296261ca1af174f24225171fea9664ba9003cbebee704810528" +dependencies = [ + "num-traits", +] + [[package]] name = "atomic-waker" version = "1.1.2" @@ -453,6 +650,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dad887fd958be91b5098c0248def011f4523ab786cd411be668777e55063501f" dependencies = [ "find-msvc-tools", + "jobserver", + "libc", "shlex", ] @@ -460,6 +659,7 @@ dependencies = [ name = "ceesvee" version = "0.4.0" dependencies = [ + "arrow", "chardetng", "chrono", "chrono-tz", @@ -468,6 +668,7 @@ dependencies = [ "flate2", "getrandom 0.2.17", "hmac", + "parquet", "regex", "serde", "serde_json", @@ -581,6 +782,26 @@ dependencies = [ "crossbeam-utils", ] +[[package]] +name = "const-random" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" +dependencies = [ + "const-random-macro", +] + +[[package]] +name = "const-random-macro" +version = "0.1.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" +dependencies = [ + "getrandom 0.2.17", + "once_cell", + "tiny-keccak", +] + [[package]] name = "cookie" version = "0.18.1" @@ -1139,6 +1360,16 @@ version = "0.5.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" +[[package]] +name = "flatbuffers" +version = "25.12.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3" +dependencies = [ + "bitflags 2.13.0", + "rustc_version", +] + [[package]] name = "flate2" version = "1.1.9" @@ -1598,6 +1829,7 @@ checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" dependencies = [ "cfg-if", "crunchy", + "num-traits", "zerocopy", ] @@ -2085,6 +2317,16 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "jobserver" +version = "0.1.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" +dependencies = [ + "getrandom 0.4.2", + "libc", +] + [[package]] name = "js-sys" version = "0.3.102" @@ -2135,6 +2377,63 @@ version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" +[[package]] +name = "lexical-core" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d8d125a277f807e55a77304455eb7b1cb52f2b18c143b60e766c120bd64a594" +dependencies = [ + "lexical-parse-float", + "lexical-parse-integer", + "lexical-util", + "lexical-write-float", + "lexical-write-integer", +] + +[[package]] +name = "lexical-parse-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56" +dependencies = [ + "lexical-parse-integer", + "lexical-util", +] + +[[package]] +name = "lexical-parse-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34" +dependencies = [ + "lexical-util", +] + +[[package]] +name = "lexical-util" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17" + +[[package]] +name = "lexical-write-float" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50c438c87c013188d415fbabbb1dceb44249ab81664efbd31b14ae55dabb6361" +dependencies = [ + "lexical-util", + "lexical-write-integer", +] + +[[package]] +name = "lexical-write-integer" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "409851a618475d2d5796377cad353802345cba92c867d9fbcde9cf4eac4e14df" +dependencies = [ + "lexical-util", +] + [[package]] name = "libappindicator" version = "0.9.0" @@ -2184,6 +2483,12 @@ dependencies = [ "winapi", ] +[[package]] +name = "libm" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" + [[package]] name = "libredox" version = "0.1.17" @@ -2349,12 +2654,40 @@ dependencies = [ "memchr", ] +[[package]] +name = "num-bigint" +version = "0.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + [[package]] name = "num-conv" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + [[package]] name = "num-traits" version = "0.2.19" @@ -2362,6 +2695,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" dependencies = [ "autocfg", + "libm", ] [[package]] @@ -2713,6 +3047,40 @@ dependencies = [ "windows-link 0.2.1", ] +[[package]] +name = "parquet" +version = "59.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5302d4da74d6596a1f11f9928767995b53bca657cbeea1e4e8c5074f8a1157dd" +dependencies = [ + "ahash", + "arrow-array", + "arrow-buffer", + "arrow-data", + "arrow-ipc", + "arrow-schema", + "arrow-select", + "base64 0.22.1", + "bytes", + "chrono", + "half", + "hashbrown 0.17.1", + "num-bigint", + "num-integer", + "num-traits", + "paste", + "seq-macro", + "snap", + "twox-hash", + "zstd", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + [[package]] name = "pathdiff" version = "0.2.3" @@ -3410,6 +3778,12 @@ dependencies = [ "serde_core", ] +[[package]] +name = "seq-macro" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" + [[package]] name = "serde" version = "1.0.228" @@ -3635,6 +4009,12 @@ version = "1.15.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" +[[package]] +name = "snap" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "199905e6153d6405f9728fe44daace35f8f837bbf830bb6e85fbd5828709a886" + [[package]] name = "socket2" version = "0.6.4" @@ -4341,6 +4721,15 @@ dependencies = [ "time-core", ] +[[package]] +name = "tiny-keccak" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" +dependencies = [ + "crunchy", +] + [[package]] name = "tinystr" version = "0.8.3" @@ -4638,6 +5027,12 @@ version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" +[[package]] +name = "twox-hash" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" + [[package]] name = "typeid" version = "1.0.3" @@ -6037,6 +6432,34 @@ dependencies = [ "simd-adler32", ] +[[package]] +name = "zstd" +version = "0.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" +dependencies = [ + "zstd-safe", +] + +[[package]] +name = "zstd-safe" +version = "7.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" +dependencies = [ + "zstd-sys", +] + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] + [[package]] name = "zune-core" version = "0.5.1" diff --git a/src-tauri/Cargo.toml b/src-tauri/Cargo.toml index 41fc572..17a5b4a 100644 --- a/src-tauri/Cargo.toml +++ b/src-tauri/Cargo.toml @@ -52,6 +52,9 @@ getrandom = "0.2" # Compressed CSV support (F17): streaming gzip + ZIP archives. flate2 = "1" zip = { version = "2", default-features = false, features = ["deflate"] } +# Parquet & Arrow interop (F32): typed columnar open/export. +arrow = { version = "59", default-features = false, features = ["ipc"] } +parquet = { version = "59", default-features = false, features = ["arrow", "snap", "zstd"] } # Single-instance is desktop-only: it forwards a second "Open with" launch # (and its file argument) to the already-running window instead of spawning a diff --git a/src-tauri/src/columnar_export.rs b/src-tauri/src/columnar_export.rs new file mode 100644 index 0000000..ab0f3ab --- /dev/null +++ b/src-tauri/src/columnar_export.rs @@ -0,0 +1,1442 @@ +//! Parquet & Arrow interop (F32) — write side. +//! +//! Exports a document — ANY export scope from [`crate::export_scope`] (all +//! rows, the filtered view, selected rows/columns/range) — to Apache Parquet +//! (uncompressed / Snappy / Zstd), an Arrow IPC file (= Feather v2; keep the +//! alias in UI copy) or an Arrow IPC stream, through the F03 atomic-save +//! pipeline as a cancellable job. +//! +//! Typing rules (`typed`, the default): +//! - A column WITH a declared F31 schema exports as the arrow type its +//! logical type maps to: integer → Int64 (or UInt64 when the scoped values +//! include one beyond `i64::MAX` and none negative — a typing pre-pass over +//! the scoped rows decides); decimal → Decimal128 with the smallest +//! precision/scale covering every valid scoped cell (per-cell scales unify +//! to the widest EXACTLY — `1.5` in a scale-2 column exports as mantissa +//! `150`; a column needing more than 38 fractional digits falls back to +//! Utf8); float → Float64; boolean → Boolean; date → Date32; datetime → +//! Timestamp in microseconds — or nanoseconds when any scoped value +//! carries sub-microsecond digits — with [`ColumnSchema::time_zone`] +//! preserved as the arrow timezone (values are the UTC instants the schema +//! parse already produces); text/uuid/json → Utf8, cell text verbatim. +//! - Schema null tokens export as columnar NULL. An empty cell in a +//! non-text typed column is a null WITHOUT a warning (it means "no value", +//! mirroring [`crate::schema::classify`]'s `Empty` state). +//! - A cell that fails to parse under its declared schema — or whose value +//! the chosen arrow type cannot represent (`u64::MAX` in a column that +//! also holds negatives, a decimal wider than the unified precision, a +//! sub-microsecond timestamp outside the nanosecond range) — exports as +//! NULL and counts into the per-column totals on the returned +//! [`ColumnarExportReport`]. Nothing is ever substituted silently. +//! - Without a schema (or with `typed: false`) every column exports as Utf8 +//! text exactly as stored, so `007` stays `007`. +//! +//! Null vs empty string: scoped rows are read through the `Option` plane +//! ([`crate::tabular`] semantics — a columnar-backed document's NULL arrives +//! as `None`, an empty string as `Some("")`), so both survive distinctly: +//! `None` → columnar NULL, `Some("")` → an empty Utf8 string. Editable and +//! F10-indexed documents never produce `None`; their nulls are schema null +//! tokens. +//! +//! Compression applies to Parquet only (Arrow IPC always writes +//! uncompressed; the option is ignored there), and `row_group_rows` bounds +//! parquet row-group sizes so the read side's statistics pruning has groups +//! to skip. +//! +//! Cancellation/atomicity: the typing pre-pass and every batch write observe +//! the [`JobCtx`]; all bytes stream into the F03 staging file, so failure or +//! cancellation at any point removes the staging file and never touches an +//! existing destination. + +use std::collections::HashMap; +use std::fs::File; +use std::io::Write; +use std::path::Path; +use std::sync::{Arc, Mutex}; + +use arrow::array::{ + ArrayRef, BooleanBuilder, Date32Builder, Decimal128Builder, Float64Builder, Int64Builder, + StringBuilder, TimestampMicrosecondBuilder, TimestampNanosecondBuilder, UInt64Builder, +}; +use arrow::datatypes::{DataType, Field, Schema as ArrowSchema, SchemaRef, TimeUnit}; +use arrow::ipc::writer::{FileWriter as IpcFileWriter, StreamWriter as IpcStreamWriter}; +use arrow::record_batch::RecordBatch; +use chrono::{NaiveDate, Timelike}; +use parquet::arrow::ArrowWriter; +use parquet::basic::{Compression, ZstdLevel}; +use parquet::file::properties::WriterProperties; +use serde::{Deserialize, Serialize}; + +use crate::document::Document; +use crate::dto::{BackupPolicy, ExportScope}; +use crate::error::{AppError, AppResult}; +use crate::export_scope::{self, ResolvedScope}; +use crate::job::JobCtx; +use crate::parquet_arrow::ColumnarFormat; +use crate::save; +use crate::schema::{ + self, classify, CellState, ColumnSchema, DecimalValue, LogicalType, TypedValue, +}; +use crate::tabular::TabularRow; + +/// Rows per written record batch (matches the read side's decode block, so +/// an export → re-open round trip stays block-aligned). +const BATCH_ROWS: usize = 4096; + +fn write_err(e: impl std::fmt::Display) -> AppError { + AppError::Other(format!("columnar write error: {e}")) +} + +// --------------------------------------------------------------------------- +// Options / report DTOs +// --------------------------------------------------------------------------- + +/// Parquet compression codec choice (wire DTO, camelCase). Applies to the +/// Parquet format only; Arrow IPC output is always uncompressed. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum ColumnarCompression { + Uncompressed, + #[default] + Snappy, + Zstd, +} + +/// Options for [`run`] (wire DTO, camelCase). +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase", default)] +pub struct ColumnarExportOptions { + /// Output container: Parquet, Arrow IPC file (Feather v2) or stream. + pub format: ColumnarFormat, + /// Parquet compression codec (ignored for the Arrow IPC formats). + pub compression: ColumnarCompression, + /// Emit typed arrow columns for columns with a declared F31 schema + /// (module docs). `false` = every column as Utf8 text. + pub typed: bool, + /// Parquet only: maximum rows per row group (`0` = writer default). + /// Smaller groups give the read side's statistics pruning more to skip. + pub row_group_rows: usize, + /// Backup policy for the previous destination file. + pub backup: BackupPolicy, +} + +impl Default for ColumnarExportOptions { + fn default() -> ColumnarExportOptions { + ColumnarExportOptions { + format: ColumnarFormat::Parquet, + compression: ColumnarCompression::default(), + typed: true, + row_group_rows: 0, + backup: BackupPolicy::default(), + } + } +} + +/// Per-column invalid-cell total on a finished export (wire DTO, camelCase). +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ColumnWarning { + /// Output column name. + pub name: String, + /// Cells that exported as NULL because they could not be represented + /// under the declared schema / chosen arrow type. + pub invalid_cells: u64, +} + +/// What a finished export produced (wire DTO, camelCase). Fetched by job id +/// after the `job-finished` event. +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ColumnarExportReport { + /// Wire name of the container written (`parquet` / `arrowFile` / + /// `arrowStream`). + pub format: String, + /// Data rows written. + pub rows: u64, + /// Columns written. + pub columns: usize, + /// Bytes written to the destination. + pub bytes: u64, + /// Total cells exported as NULL with a warning (module docs). + pub invalid_cells: u64, + /// Per-column breakdown (only columns with at least one warning). + pub column_warnings: Vec, +} + +/// Finished export reports keyed by the job id that produced them (mirrors +/// the JSON-import preview cache). +#[derive(Default)] +pub struct ColumnarExportReportCache(Arc>>); + +impl ColumnarExportReportCache { + pub fn share(&self) -> Arc>> { + Arc::clone(&self.0) + } + + pub fn get(&self, job_id: u64) -> Option { + self.0.lock().ok()?.get(&job_id).cloned() + } +} + +// --------------------------------------------------------------------------- +// Planning: scope + per-column arrow targets +// --------------------------------------------------------------------------- + +/// The arrow type one output column writes. +#[derive(Debug, Clone, PartialEq)] +enum Target { + Utf8, + Int64, + UInt64, + Float64, + Decimal { precision: u8, scale: u32 }, + Boolean, + Date32, + Timestamp { nanos: bool, tz: Option }, +} + +/// One planned output column (parallel to `resolved.cols`, which carries +/// the absolute document column index). +struct PlannedColumn { + name: String, + /// The declared schema driving typed emission (`None` = plain Utf8). + schema: Option, + target: Target, +} + +struct ExportPlan { + resolved: ResolvedScope, + cols: Vec, +} + +/// Typing pre-pass aggregates for one column. +#[derive(Debug, Default)] +struct Probe { + any_negative: bool, + any_over_i64: bool, + max_scale: u32, + max_int_digits: usize, + subsec: bool, +} + +/// Resolve the scope and decide every column's arrow target. Columns whose +/// target depends on the data (integer width, decimal precision/scale, +/// timestamp unit) are decided by ONE typing pre-pass over the scoped rows; +/// everything else is decided from the schema alone. Sets the job total +/// (rows × passes) as a side effect. +fn plan( + doc: &Document, + options: &ColumnarExportOptions, + scope: &ExportScope, + ctx: &JobCtx, +) -> AppResult { + let resolved = export_scope::resolve_scope(doc, scope)?; + let headers = doc.headers(); + let mut cols: Vec = Vec::with_capacity(resolved.cols.len()); + let mut probes: Vec> = Vec::with_capacity(resolved.cols.len()); + for &c in &resolved.cols { + let schema = if options.typed { + doc.column_schema_at(c).cloned() + } else { + None + }; + let (target, probe) = match schema.as_ref().map(|s| s.logical_type) { + Some(LogicalType::Integer) => (Target::Int64, Some(Probe::default())), + Some(LogicalType::Decimal) => ( + Target::Decimal { + precision: 1, + scale: 0, + }, + Some(Probe::default()), + ), + Some(LogicalType::Float) => (Target::Float64, None), + Some(LogicalType::Boolean) => (Target::Boolean, None), + Some(LogicalType::Date) => (Target::Date32, None), + Some(LogicalType::Datetime) => ( + Target::Timestamp { + nanos: false, + tz: schema.as_ref().and_then(|s| s.time_zone.clone()), + }, + Some(Probe::default()), + ), + // Text, Uuid and Json stay text, verbatim; no schema = text. + _ => (Target::Utf8, None), + }; + cols.push(PlannedColumn { + name: headers[c].clone(), + schema, + target, + }); + probes.push(probe); + } + + let needs_prepass = probes.iter().any(Option::is_some); + let passes = 1 + u64::from(needs_prepass); + ctx.set_total(resolved.rows.len() as u64 * passes); + + if needs_prepass { + stream_scoped(doc, &resolved, ctx, |chunk| { + for row in chunk { + for (i, probe) in probes.iter_mut().enumerate() { + let Some(probe) = probe.as_mut() else { + continue; + }; + let Some(text) = row[i].as_deref() else { + continue; + }; + let Some(schema) = cols[i].schema.as_ref() else { + continue; + }; + let CellState::Valid(value) = classify(Some(text), schema) else { + continue; + }; + match value { + TypedValue::Integer(v) => { + if v < 0 { + probe.any_negative = true; + } + if v > i128::from(i64::MAX) { + probe.any_over_i64 = true; + } + } + TypedValue::Decimal(d) => { + probe.max_scale = probe.max_scale.max(d.scale); + let int_digits = d.digits.len().saturating_sub(d.scale as usize); + probe.max_int_digits = probe.max_int_digits.max(int_digits); + } + TypedValue::DateTime(ndt) if ndt.nanosecond() % 1000 != 0 => { + probe.subsec = true; + } + _ => {} + } + } + } + Ok(()) + })?; + for (planned, probe) in cols.iter_mut().zip(&probes) { + let Some(probe) = probe else { continue }; + planned.target = match &planned.target { + Target::Int64 => { + if probe.any_over_i64 && !probe.any_negative { + Target::UInt64 + } else { + Target::Int64 + } + } + Target::Decimal { .. } => { + let scale = probe.max_scale; + if scale > 38 { + // Decimal128 cannot carry the fractional width; keep + // the exact text instead of rounding (module docs). + Target::Utf8 + } else { + let precision = (probe.max_int_digits as u32 + scale).clamp(1, 38) as u8; + Target::Decimal { precision, scale } + } + } + Target::Timestamp { tz, .. } => Target::Timestamp { + nanos: probe.subsec, + tz: tz.clone(), + }, + other => other.clone(), + }; + } + } + + Ok(ExportPlan { resolved, cols }) +} + +fn field_of(planned: &PlannedColumn) -> Field { + let data_type = match &planned.target { + Target::Utf8 => DataType::Utf8, + Target::Int64 => DataType::Int64, + Target::UInt64 => DataType::UInt64, + Target::Float64 => DataType::Float64, + Target::Decimal { precision, scale } => DataType::Decimal128(*precision, *scale as i8), + Target::Boolean => DataType::Boolean, + Target::Date32 => DataType::Date32, + Target::Timestamp { nanos, tz } => DataType::Timestamp( + if *nanos { + TimeUnit::Nanosecond + } else { + TimeUnit::Microsecond + }, + tz.clone().map(Arc::from), + ), + }; + Field::new(planned.name.clone(), data_type, true) +} + +// --------------------------------------------------------------------------- +// Scoped Option-plane streaming +// --------------------------------------------------------------------------- + +/// Stream the resolved rows, projected to the resolved columns, as bounded +/// chunks of `Option` cells. Columnar-backed documents read the handle's +/// `Option` plane (NULL = `None`) with consecutive scoped rows coalesced +/// into shared windowed reads; the text backings never produce `None`. +/// Advances the job by every row delivered (cancellation is observed there). +fn stream_scoped( + doc: &Document, + resolved: &ResolvedScope, + ctx: &JobCtx, + mut per_chunk: impl FnMut(Vec) -> AppResult<()>, +) -> AppResult<()> { + let cols = &resolved.cols; + let mut buffer: Vec = Vec::with_capacity(BATCH_ROWS.min(resolved.rows.len())); + if let Some(handle) = doc.columnar_handle() { + let rows = &resolved.rows; + let mut i = 0usize; + while i < rows.len() { + // Coalesce a run of consecutive absolute rows into one read. + let start = rows[i]; + let mut len = 1usize; + while i + len < rows.len() && rows[i + len] == start + len && len < BATCH_ROWS { + len += 1; + } + let batch = handle.read_optional(start as u64, len, Some(ctx))?; + if batch.len() != len { + return Err(AppError::Other( + "the source file changed on disk; reload the document".into(), + )); + } + for row in batch { + buffer.push( + cols.iter() + .map(|&c| row.get(c).cloned().flatten()) + .collect(), + ); + if buffer.len() >= BATCH_ROWS { + ctx.advance(buffer.len() as u64)?; + per_chunk(std::mem::take(&mut buffer))?; + } + } + i += len; + } + } else { + doc.visit_rows_at(&resolved.rows, &mut |_, row| { + buffer.push(cols.iter().map(|&c| Some(row[c].clone())).collect()); + if buffer.len() >= BATCH_ROWS { + ctx.advance(buffer.len() as u64)?; + per_chunk(std::mem::take(&mut buffer))?; + } + Ok(true) + })?; + } + if !buffer.is_empty() { + ctx.advance(buffer.len() as u64)?; + per_chunk(buffer)?; + } + Ok(()) +} + +// --------------------------------------------------------------------------- +// Cell → arrow value conversion +// --------------------------------------------------------------------------- + +enum ColBuilder { + Utf8(StringBuilder), + Int64(Int64Builder), + UInt64(UInt64Builder), + Float64(Float64Builder), + Decimal(Decimal128Builder), + Boolean(BooleanBuilder), + Date32(Date32Builder), + TsMicro(TimestampMicrosecondBuilder), + TsNano(TimestampNanosecondBuilder), +} + +fn builder_for(target: &Target) -> ColBuilder { + match target { + Target::Utf8 => ColBuilder::Utf8(StringBuilder::new()), + Target::Int64 => ColBuilder::Int64(Int64Builder::new()), + Target::UInt64 => ColBuilder::UInt64(UInt64Builder::new()), + Target::Float64 => ColBuilder::Float64(Float64Builder::new()), + Target::Decimal { .. } => ColBuilder::Decimal(Decimal128Builder::new()), + Target::Boolean => ColBuilder::Boolean(BooleanBuilder::new()), + Target::Date32 => ColBuilder::Date32(Date32Builder::new()), + Target::Timestamp { nanos: false, .. } => { + ColBuilder::TsMicro(TimestampMicrosecondBuilder::new()) + } + Target::Timestamp { nanos: true, .. } => { + ColBuilder::TsNano(TimestampNanosecondBuilder::new()) + } + } +} + +impl ColBuilder { + fn append_null(&mut self) { + match self { + ColBuilder::Utf8(b) => b.append_null(), + ColBuilder::Int64(b) => b.append_null(), + ColBuilder::UInt64(b) => b.append_null(), + ColBuilder::Float64(b) => b.append_null(), + ColBuilder::Decimal(b) => b.append_null(), + ColBuilder::Boolean(b) => b.append_null(), + ColBuilder::Date32(b) => b.append_null(), + ColBuilder::TsMicro(b) => b.append_null(), + ColBuilder::TsNano(b) => b.append_null(), + } + } +} + +/// The exact Decimal128 mantissa of `d` rescaled to `scale` fraction digits, +/// when it fits (`None` = wider than i128, defensively also a cell whose own +/// scale exceeds the unified one). +fn decimal_mantissa(d: &DecimalValue, scale: u32) -> Option { + if d.scale > scale { + return None; + } + let mut mantissa: i128 = d.digits.parse().ok()?; + for _ in 0..(scale - d.scale) { + mantissa = mantissa.checked_mul(10)?; + } + Some(if d.negative { -mantissa } else { mantissa }) +} + +/// Append one typed value to its builder. `Err(())` = the chosen arrow type +/// cannot represent it (the caller writes null + counts a warning). +fn append_typed(builder: &mut ColBuilder, target: &Target, value: TypedValue) -> Result<(), ()> { + match (builder, value) { + (ColBuilder::Int64(b), TypedValue::Integer(v)) => match i64::try_from(v) { + Ok(v) => { + b.append_value(v); + Ok(()) + } + Err(_) => Err(()), + }, + (ColBuilder::UInt64(b), TypedValue::Integer(v)) => match u64::try_from(v) { + Ok(v) => { + b.append_value(v); + Ok(()) + } + Err(_) => Err(()), + }, + (ColBuilder::Float64(b), TypedValue::Float(v)) => { + b.append_value(v); + Ok(()) + } + (ColBuilder::Decimal(b), TypedValue::Decimal(d)) => { + let Target::Decimal { precision, scale } = target else { + return Err(()); + }; + let mantissa = decimal_mantissa(&d, *scale).ok_or(())?; + // 10^precision fits i128 for every legal precision (≤ 38). + if mantissa.abs() >= 10i128.pow(u32::from(*precision)) { + return Err(()); + } + b.append_value(mantissa); + Ok(()) + } + (ColBuilder::Boolean(b), TypedValue::Boolean(v)) => { + b.append_value(v); + Ok(()) + } + (ColBuilder::Date32(b), TypedValue::Date(d)) => { + let epoch = NaiveDate::from_ymd_opt(1970, 1, 1).expect("epoch is valid"); + match i32::try_from((d - epoch).num_days()) { + Ok(days) => { + b.append_value(days); + Ok(()) + } + Err(_) => Err(()), + } + } + // Parse produced a UTC instant for zoned columns and a naive wall + // time otherwise; arrow stores epoch ticks either way. The full + // NaiveDateTime range fits i64 microseconds, so the µs arm is total. + (ColBuilder::TsMicro(b), TypedValue::DateTime(ndt)) => { + b.append_value(ndt.and_utc().timestamp_micros()); + Ok(()) + } + (ColBuilder::TsNano(b), TypedValue::DateTime(ndt)) => { + match ndt.and_utc().timestamp_nanos_opt() { + Some(nanos) => { + b.append_value(nanos); + Ok(()) + } + None => Err(()), + } + } + // classify() always yields the variant matching the logical type; + // this arm is defensive. + _ => Err(()), + } +} + +/// Append one `Option` cell. Returns the number of warnings (0 or 1). +fn append_cell(builder: &mut ColBuilder, planned: &PlannedColumn, cell: Option<&str>) -> u64 { + let Some(text) = cell else { + // Columnar NULL from the Option plane. + builder.append_null(); + return 0; + }; + if let ColBuilder::Utf8(b) = builder { + // Text stays verbatim; only a declared null token becomes NULL, so + // `Some("")` survives as an empty string, distinct from null. + match &planned.schema { + Some(schema) if schema::is_null_token(schema, text) => b.append_null(), + _ => b.append_value(text), + } + return 0; + } + let Some(schema) = planned.schema.as_ref() else { + // Non-text targets are only ever chosen from a declared schema. + builder.append_null(); + return 1; + }; + match classify(Some(text), schema) { + CellState::NullToken | CellState::Empty | CellState::Missing => { + builder.append_null(); + 0 + } + CellState::Invalid(_) => { + builder.append_null(); + 1 + } + CellState::Valid(value) => { + if append_typed(builder, &planned.target, value).is_ok() { + 0 + } else { + builder.append_null(); + 1 + } + } + } +} + +fn finish_builder(builder: ColBuilder, target: &Target) -> AppResult { + Ok(match (builder, target) { + (ColBuilder::Utf8(mut b), _) => Arc::new(b.finish()), + (ColBuilder::Int64(mut b), _) => Arc::new(b.finish()), + (ColBuilder::UInt64(mut b), _) => Arc::new(b.finish()), + (ColBuilder::Float64(mut b), _) => Arc::new(b.finish()), + (ColBuilder::Decimal(mut b), Target::Decimal { precision, scale }) => Arc::new( + b.finish() + .with_precision_and_scale(*precision, *scale as i8) + .map_err(write_err)?, + ), + (ColBuilder::Boolean(mut b), _) => Arc::new(b.finish()), + (ColBuilder::Date32(mut b), _) => Arc::new(b.finish()), + (ColBuilder::TsMicro(mut b), Target::Timestamp { tz, .. }) => { + Arc::new(b.finish().with_timezone_opt(tz.clone())) + } + (ColBuilder::TsNano(mut b), Target::Timestamp { tz, .. }) => { + Arc::new(b.finish().with_timezone_opt(tz.clone())) + } + _ => { + return Err(AppError::Other( + "internal columnar export error: builder/target mismatch".into(), + )) + } + }) +} + +// --------------------------------------------------------------------------- +// Writing +// --------------------------------------------------------------------------- + +/// Byte-counting writer feeding the job's bytes-written progress. +struct CountingWriter<'a> { + inner: &'a mut File, + bytes: u64, + ctx: &'a JobCtx, +} + +impl Write for CountingWriter<'_> { + fn write(&mut self, buf: &[u8]) -> std::io::Result { + let n = self.inner.write(buf)?; + self.bytes += n as u64; + self.ctx.add_bytes(n as u64); + Ok(n) + } + + fn flush(&mut self) -> std::io::Result<()> { + self.inner.flush() + } +} + +fn compression_of(c: ColumnarCompression) -> Compression { + match c { + ColumnarCompression::Uncompressed => Compression::UNCOMPRESSED, + ColumnarCompression::Snappy => Compression::SNAPPY, + ColumnarCompression::Zstd => Compression::ZSTD(ZstdLevel::default()), + } +} + +/// Stream the scoped rows as record batches into `write`. +fn write_batches( + doc: &Document, + plan: &ExportPlan, + schema: &SchemaRef, + warnings: &mut [u64], + ctx: &JobCtx, + mut write: impl FnMut(&RecordBatch) -> AppResult<()>, +) -> AppResult<()> { + stream_scoped(doc, &plan.resolved, ctx, |chunk| { + let mut builders: Vec = + plan.cols.iter().map(|c| builder_for(&c.target)).collect(); + for row in &chunk { + for (i, planned) in plan.cols.iter().enumerate() { + warnings[i] += append_cell(&mut builders[i], planned, row[i].as_deref()); + } + } + let arrays: Vec = builders + .into_iter() + .zip(&plan.cols) + .map(|(b, c)| finish_builder(b, &c.target)) + .collect::>()?; + let batch = RecordBatch::try_new(schema.clone(), arrays).map_err(write_err)?; + write(&batch) + }) +} + +/// Run a revision-guarded, cancellable scoped export to Parquet / Arrow IPC +/// through the atomic-save pipeline. Any failure — stale revision, I/O, +/// cancellation — leaves the destination byte-for-byte untouched and removes +/// the staging file. +pub fn run( + doc: &Document, + dest: &Path, + options: &ColumnarExportOptions, + scope: &ExportScope, + expected_revision: u64, + ctx: &JobCtx, +) -> AppResult { + doc.check_revision(expected_revision)?; + let plan = plan(doc, options, scope, ctx)?; + let fields: Vec = plan.cols.iter().map(field_of).collect(); + let schema: SchemaRef = Arc::new(ArrowSchema::new(fields)); + let mut warnings = vec![0u64; plan.cols.len()]; + + let bytes = save::atomic_write(dest, options.backup, |file| { + let mut counting = CountingWriter { + inner: file, + bytes: 0, + ctx, + }; + match options.format { + ColumnarFormat::Parquet => { + let mut props = WriterProperties::builder() + .set_compression(compression_of(options.compression)); + if options.row_group_rows > 0 { + props = props.set_max_row_group_row_count(Some(options.row_group_rows)); + } + let mut writer = + ArrowWriter::try_new(&mut counting, schema.clone(), Some(props.build())) + .map_err(write_err)?; + write_batches(doc, &plan, &schema, &mut warnings, ctx, |batch| { + writer.write(batch).map_err(write_err) + })?; + writer.close().map_err(write_err)?; + } + ColumnarFormat::ArrowFile => { + let mut writer = + IpcFileWriter::try_new(&mut counting, schema.as_ref()).map_err(write_err)?; + write_batches(doc, &plan, &schema, &mut warnings, ctx, |batch| { + writer.write(batch).map_err(write_err) + })?; + writer.finish().map_err(write_err)?; + } + ColumnarFormat::ArrowStream => { + let mut writer = + IpcStreamWriter::try_new(&mut counting, schema.as_ref()).map_err(write_err)?; + write_batches(doc, &plan, &schema, &mut warnings, ctx, |batch| { + writer.write(batch).map_err(write_err) + })?; + writer.finish().map_err(write_err)?; + } + } + Ok(counting.bytes) + })?; + ctx.flush_progress(); + + let column_warnings: Vec = plan + .cols + .iter() + .zip(&warnings) + .filter(|(_, &w)| w > 0) + .map(|(c, &w)| ColumnWarning { + name: c.name.clone(), + invalid_cells: w, + }) + .collect(); + Ok(ColumnarExportReport { + format: options.format.wire_name().to_string(), + rows: plan.resolved.rows.len() as u64, + columns: plan.cols.len(), + bytes, + invalid_cells: warnings.iter().sum(), + column_warnings, + }) +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +#[cfg(test)] +mod tests { + use super::*; + + use arrow::array::{Decimal128Array, StringArray, TimestampMicrosecondArray}; + + use crate::dto::{Conjunction, FilterCondition, FilterGroup, FilterNode, FilterOp}; + use crate::job::JobRegistry; + use crate::parquet_arrow::{self, ColumnarOpenOptions}; + use crate::parse::{parse, ParseSettings}; + + fn doc_from(csv: &str, has_header: bool) -> Document { + let parsed = parse(csv.as_bytes(), &ParseSettings::default()).unwrap(); + Document::from_parsed(1, None, parsed, has_header) + } + + /// Attach a schema (by position) to an editable document. + fn set_schema( + doc: &mut Document, + col: usize, + lt: LogicalType, + tweak: impl FnOnce(&mut ColumnSchema), + ) { + let id = doc.column_ids()[col].clone(); + let name = doc.headers()[col].clone(); + let mut schema = ColumnSchema::new(id, name, lt); + tweak(&mut schema); + doc.set_column_schema(schema); + } + + fn ctx() -> (JobRegistry, JobCtx) { + let registry = JobRegistry::default(); + let ctx = registry.begin("export", Some(1), |_| {}); + (registry, ctx) + } + + fn options(format: ColumnarFormat) -> ColumnarExportOptions { + ColumnarExportOptions { + format, + ..ColumnarExportOptions::default() + } + } + + fn export_ok( + doc: &Document, + dest: &Path, + options: &ColumnarExportOptions, + scope: &ExportScope, + ) -> ColumnarExportReport { + let (_r, ctx) = ctx(); + run(doc, dest, options, scope, doc.revision(), &ctx).unwrap() + } + + fn reopen(path: &Path) -> parquet_arrow::ColumnarFile { + parquet_arrow::open_indexed(path, &ColumnarOpenOptions::default(), None).unwrap() + } + + fn optional_plane(file: &parquet_arrow::ColumnarFile) -> Vec { + file.handle + .read_optional(0, file.handle.n_rows(), None) + .unwrap() + } + + fn text_plane(file: &parquet_arrow::ColumnarFile) -> Vec> { + let mut out = Vec::new(); + file.handle + .visit(0..file.handle.n_rows(), &mut |_, row| { + out.push(row.to_vec()); + Ok(true) + }) + .unwrap(); + out + } + + /// Write a one-batch parquet file directly through arrow (test input). + fn write_parquet(path: &Path, cols: Vec<(&str, ArrayRef)>) { + let fields: Vec = cols + .iter() + .map(|(n, a)| Field::new(*n, a.data_type().clone(), true)) + .collect(); + let schema = Arc::new(ArrowSchema::new(fields)); + let batch = + RecordBatch::try_new(schema.clone(), cols.into_iter().map(|(_, a)| a).collect()) + .unwrap(); + let file = File::create(path).unwrap(); + let mut writer = ArrowWriter::try_new(file, schema, None).unwrap(); + writer.write(&batch).unwrap(); + writer.close().unwrap(); + } + + fn some(s: &str) -> Option { + Some(s.to_string()) + } + + // ----- null vs empty --------------------------------------------------- + + #[test] + fn null_token_and_empty_string_round_trip_distinctly() { + // A second column keeps every row non-blank (the CSV parser skips + // fully blank lines). + let mut d = doc_from("s,k\nNULL,a\n,b\nx,c", true); + set_schema(&mut d, 0, LogicalType::Text, |s| { + s.null_tokens = vec!["NULL".into()]; + }); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!(report.rows, 3); + assert_eq!(report.invalid_cells, 0); + + let file = reopen(&dest); + assert_eq!( + optional_plane(&file), + vec![ + vec![None, some("a")], + vec![some(""), some("b")], + vec![some("x"), some("c")], + ], + "null token -> NULL; empty string stays an empty string" + ); + } + + #[test] + fn columnar_document_re_export_preserves_null_vs_empty() { + // A columnar-backed document's Option plane is the source of truth: + // NULL arrives as None and must survive a re-export without any null + // token in play. + let dir = tempfile::tempdir().unwrap(); + let source = dir.path().join("in.parquet"); + write_parquet( + &source, + vec![( + "s", + Arc::new(StringArray::from(vec![None, Some(""), Some("x")])) as ArrayRef, + )], + ); + let doc = Document::from_columnar(1, Some(source.clone()), reopen(&source)); + + let dest = dir.path().join("out.parquet"); + export_ok( + &doc, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + let file = reopen(&dest); + assert_eq!( + optional_plane(&file), + vec![vec![None], vec![some("")], vec![some("x")]] + ); + } + + // ----- integers -------------------------------------------------------- + + #[test] + fn signed_integer_extremes_round_trip_as_int64() { + let mut d = doc_from("i\n9223372036854775807\n-9223372036854775808\n0", true); + set_schema(&mut d, 0, LogicalType::Integer, |_| {}); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!(report.invalid_cells, 0); + + let file = reopen(&dest); + assert_eq!(file.schemas[0].logical_type, LogicalType::Integer); + let text = text_plane(&file); + assert_eq!(text[0][0], i64::MAX.to_string()); + assert_eq!(text[1][0], i64::MIN.to_string()); + assert_eq!(text[2][0], "0"); + } + + #[test] + fn u64_max_selects_uint64_and_round_trips_losslessly() { + let mut d = doc_from("u\n18446744073709551615\n5", true); + set_schema(&mut d, 0, LogicalType::Integer, |_| {}); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!(report.invalid_cells, 0); + + let file = reopen(&dest); + let text = text_plane(&file); + assert_eq!(text[0][0], u64::MAX.to_string(), "u64::MAX lossless"); + assert_eq!(text[1][0], "5"); + } + + #[test] + fn beyond_i64_with_negatives_warns_and_nulls() { + // A single arrow integer type cannot carry BOTH u64::MAX and a + // negative; the negative keeps Int64 and the oversized value becomes + // null + a counted warning. + let mut d = doc_from("i\n18446744073709551615\n-1", true); + set_schema(&mut d, 0, LogicalType::Integer, |_| {}); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!(report.invalid_cells, 1); + assert_eq!(report.column_warnings.len(), 1); + assert_eq!(report.column_warnings[0].name, "i"); + + let file = reopen(&dest); + assert_eq!( + optional_plane(&file), + vec![vec![None], vec![some("-1")]], + "the unrepresentable value exported as NULL" + ); + } + + #[test] + fn unparseable_cells_export_null_with_warning_count() { + let mut d = doc_from("n\nabc\n5\nxyz", true); + set_schema(&mut d, 0, LogicalType::Integer, |_| {}); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!(report.invalid_cells, 2); + assert_eq!(report.column_warnings[0].invalid_cells, 2); + + let file = reopen(&dest); + assert_eq!( + optional_plane(&file), + vec![vec![None], vec![some("5")], vec![None]] + ); + } + + #[test] + fn empty_cell_in_numeric_column_is_null_without_warning() { + let mut d = doc_from("n,k\n,a\n7,b", true); + set_schema(&mut d, 0, LogicalType::Integer, |_| {}); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!( + report.invalid_cells, 0, + "empty means 'no value', not invalid" + ); + + let file = reopen(&dest); + assert_eq!( + optional_plane(&file), + vec![vec![None, some("a")], vec![some("7"), some("b")]] + ); + } + + // ----- decimals -------------------------------------------------------- + + #[test] + fn decimal_scales_unify_to_the_widest_exactly() { + let mut d = doc_from("d\n1.50\n-0.055\n2", true); + set_schema(&mut d, 0, LogicalType::Decimal, |_| {}); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!(report.invalid_cells, 0); + + let file = reopen(&dest); + assert_eq!(file.schemas[0].logical_type, LogicalType::Decimal); + let text = text_plane(&file); + assert_eq!( + ( + text[0][0].as_str(), + text[1][0].as_str(), + text[2][0].as_str() + ), + ("1.500", "-0.055", "2.000"), + "one column scale; every value rescaled exactly, never rounded" + ); + } + + #[test] + fn columnar_decimal_round_trips_with_identical_scale() { + let dir = tempfile::tempdir().unwrap(); + let source = dir.path().join("in.parquet"); + write_parquet( + &source, + vec![( + "dec", + Arc::new( + Decimal128Array::from(vec![Some(150i128), Some(-5), None]) + .with_precision_and_scale(12, 2) + .unwrap(), + ) as ArrayRef, + )], + ); + let doc = Document::from_columnar(1, Some(source.clone()), reopen(&source)); + let before = optional_plane(&reopen(&source)); + + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &doc, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!(report.invalid_cells, 0); + assert_eq!( + optional_plane(&reopen(&dest)), + before, + "uniform source scale -> byte-identical decimal text (1.50 stays 1.50)" + ); + } + + // ----- timestamps ------------------------------------------------------ + + #[test] + fn zoned_timestamp_preserves_zone_metadata_and_instants() { + let dir = tempfile::tempdir().unwrap(); + let source = dir.path().join("in.parquet"); + let ticks = 1_700_000_000_123_456i64; // 2023-11-14T22:13:20.123456Z + write_parquet( + &source, + vec![( + "ts", + Arc::new( + TimestampMicrosecondArray::from(vec![Some(ticks), None]) + .with_timezone("Europe/Berlin"), + ) as ArrayRef, + )], + ); + let opened = reopen(&source); + assert_eq!( + opened.schemas[0].time_zone.as_deref(), + Some("Europe/Berlin") + ); + let before = optional_plane(&opened); + let doc = Document::from_columnar(1, Some(source.clone()), opened); + + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &doc, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!(report.invalid_cells, 0); + + let file = reopen(&dest); + assert_eq!( + file.schemas[0].time_zone.as_deref(), + Some("Europe/Berlin"), + "timezone metadata survives the round trip" + ); + assert_eq!(optional_plane(&file), before, "UTC instants identical"); + } + + #[test] + fn subsecond_timestamps_choose_nanoseconds() { + let mut d = doc_from( + "ts\n2024-01-02T03:04:05.123456789\n2024-01-02T03:04:06", + true, + ); + set_schema(&mut d, 0, LogicalType::Datetime, |s| { + s.input_formats = Some(vec!["%Y-%m-%dT%H:%M:%S%.f".into()]); + }); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!(report.invalid_cells, 0); + + let file = reopen(&dest); + let text = text_plane(&file); + assert_eq!( + text[0][0], "2024-01-02T03:04:05.123456789", + "sub-microsecond digits survive via the nanosecond unit" + ); + assert_eq!(text[1][0], "2024-01-02T03:04:06"); + } + + // ----- the remaining scalar types -------------------------------------- + + #[test] + fn float_boolean_and_date_round_trip() { + let mut d = doc_from("f,b,d\n0.1,true,2024-01-01\n-1.5,false,1969-12-31", true); + set_schema(&mut d, 0, LogicalType::Float, |_| {}); + set_schema(&mut d, 1, LogicalType::Boolean, |_| {}); + set_schema(&mut d, 2, LogicalType::Date, |_| {}); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + assert_eq!(report.invalid_cells, 0); + + let file = reopen(&dest); + let logicals: Vec = file.schemas.iter().map(|s| s.logical_type).collect(); + assert_eq!( + logicals, + [LogicalType::Float, LogicalType::Boolean, LogicalType::Date] + ); + let text = text_plane(&file); + assert_eq!(text[0], ["0.1", "true", "2024-01-01"]); + assert_eq!(text[1], ["-1.5", "false", "1969-12-31"]); + } + + // ----- untyped exports ------------------------------------------------- + + #[test] + fn no_schema_exports_all_utf8() { + let d = doc_from("a,b\n1,x\n2,", true); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + ); + + let file = reopen(&dest); + assert!(file + .schemas + .iter() + .all(|s| s.logical_type == LogicalType::Text)); + assert_eq!( + optional_plane(&file), + vec![vec![some("1"), some("x")], vec![some("2"), some("")]], + "text verbatim; the empty CSV cell stays an empty string, not null" + ); + } + + #[test] + fn typed_false_keeps_cell_text_verbatim() { + let mut d = doc_from("n\n007", true); + set_schema(&mut d, 0, LogicalType::Integer, |_| {}); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let mut opts = options(ColumnarFormat::Parquet); + opts.typed = false; + export_ok(&d, &dest, &opts, &ExportScope::All); + + let file = reopen(&dest); + assert_eq!(file.schemas[0].logical_type, LogicalType::Text); + assert_eq!(text_plane(&file)[0][0], "007", "no canonicalisation"); + } + + // ----- scopes ---------------------------------------------------------- + + #[test] + fn scoped_export_writes_exact_rows_and_columns_in_order() { + let mut d = doc_from("a,b,c\n1,2,3\n4,5,6\n7,8,9", true); + d.set_filter(vec![0, 2]).unwrap(); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let scope = ExportScope::SelectedColumns { + columns: vec![2, 0], + }; + let report = export_ok(&d, &dest, &options(ColumnarFormat::Parquet), &scope); + assert_eq!((report.rows, report.columns), (2, 2)); + + let file = reopen(&dest); + assert_eq!(file.headers, ["c", "a"], "user column order preserved"); + assert_eq!( + text_plane(&file), + vec![vec!["3", "1"], vec!["9", "7"]], + "exactly the filtered rows, in display order" + ); + } + + #[test] + fn empty_scope_writes_a_schema_only_file() { + let mut d = doc_from("a\n1", true); + d.set_filter(vec![]).unwrap(); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let report = export_ok( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::VisibleRows, + ); + assert_eq!(report.rows, 0); + + let file = reopen(&dest); + assert_eq!(file.headers, ["a"]); + assert_eq!(file.handle.n_rows(), 0); + } + + // ----- the three containers ------------------------------------------- + + #[test] + fn arrow_file_and_stream_round_trip_like_parquet() { + let mut d = doc_from("i,s\n42,x\n,\n18446744073709551615,y", true); + set_schema(&mut d, 0, LogicalType::Integer, |_| {}); + let dir = tempfile::tempdir().unwrap(); + + let mut planes = Vec::new(); + for (format, name) in [ + (ColumnarFormat::Parquet, "out.parquet"), + (ColumnarFormat::ArrowFile, "out.arrow"), + (ColumnarFormat::ArrowStream, "out.arrows"), + ] { + let dest = dir.path().join(name); + let report = export_ok(&d, &dest, &options(format), &ExportScope::All); + assert_eq!(report.format, format.wire_name()); + let inspection = parquet_arrow::inspect(&dest, None).unwrap(); + assert_eq!(inspection.format, format.wire_name(), "format sniffs back"); + assert_eq!(inspection.row_count, 3); + planes.push(optional_plane(&reopen(&dest))); + } + assert_eq!(planes[0], planes[1]); + assert_eq!(planes[1], planes[2]); + assert_eq!( + planes[0][1], + vec![None, some("")], + "empty integer cell -> NULL; empty text cell stays empty" + ); + assert_eq!(planes[0][2][0], some("18446744073709551615")); + } + + #[test] + fn parquet_compression_codecs_apply() { + let mut d = doc_from("n\n1\n2\n3", true); + set_schema(&mut d, 0, LogicalType::Integer, |_| {}); + let dir = tempfile::tempdir().unwrap(); + + for (compression, expect) in [ + (ColumnarCompression::Uncompressed, "UNCOMPRESSED"), + (ColumnarCompression::Snappy, "SNAPPY"), + (ColumnarCompression::Zstd, "ZSTD"), + ] { + let dest = dir.path().join(format!("{expect}.parquet")); + let mut opts = options(ColumnarFormat::Parquet); + opts.compression = compression; + export_ok(&d, &dest, &opts, &ExportScope::All); + let inspection = parquet_arrow::inspect(&dest, None).unwrap(); + let codecs = inspection.compression.unwrap_or_default(); + assert!(codecs.contains(expect), "{expect} not in {codecs:?}"); + assert_eq!(text_plane(&reopen(&dest))[2][0], "3"); + } + } + + // ----- row groups + statistics pruning --------------------------------- + + #[test] + fn row_group_size_applies_and_pruned_filtered_reads_match_full_scan() { + let mut csv = String::from("n\n"); + for i in 0..100 { + csv.push_str(&format!("{i}\n")); + } + let mut d = doc_from(&csv, true); + set_schema(&mut d, 0, LogicalType::Integer, |_| {}); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let mut opts = options(ColumnarFormat::Parquet); + opts.row_group_rows = 10; + export_ok(&d, &dest, &opts, &ExportScope::All); + + let inspection = parquet_arrow::inspect(&dest, None).unwrap(); + assert_eq!(inspection.chunk_count, 10, "row_group_rows bounds groups"); + + let doc = Document::from_columnar(2, Some(dest.clone()), reopen(&dest)); + let spec = FilterGroup { + conjunction: Conjunction::And, + nodes: vec![FilterNode::Condition(FilterCondition { + column: 0, + op: FilterOp::Gte, + value: "73".into(), + case_sensitive: false, + })], + }; + let ranges = doc + .filter_scan_ranges(&spec) + .expect("statistics pruning applies"); + let visited: usize = ranges.iter().map(|r| r.len()).sum(); + assert!(visited < 100, "row groups below the bound are skipped"); + + let matches = crate::filter::matching_rows(&doc, &spec).unwrap(); + assert_eq!( + matches, + (73..100).collect::>(), + "pruned filtered read returns exactly the full-scan matches" + ); + } + + // ----- guards ---------------------------------------------------------- + + #[test] + fn cancelled_export_removes_all_output() { + let d = doc_from("a\n1\n2\n3", true); + let registry = JobRegistry::default(); + let ctx = registry.begin("export", Some(1), |_| {}); + registry.cancel(ctx.id); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("cancel.parquet"); + let result = run( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + d.revision(), + &ctx, + ); + assert!(matches!(result, Err(AppError::Cancelled))); + assert!(!dest.exists(), "no destination file"); + assert_eq!( + std::fs::read_dir(dir.path()).unwrap().count(), + 0, + "no staging litter" + ); + } + + #[test] + fn stale_revision_is_rejected_before_writing() { + let mut d = doc_from("a\n1", true); + let stale = d.revision(); + d.set_cell(0, 0, "changed".into()).unwrap(); + let dir = tempfile::tempdir().unwrap(); + let dest = dir.path().join("out.parquet"); + let (_r, ctx) = ctx(); + let err = run( + &d, + &dest, + &options(ColumnarFormat::Parquet), + &ExportScope::All, + stale, + &ctx, + ) + .unwrap_err(); + assert!(matches!(err, AppError::StaleRevision { .. }), "{err}"); + assert!(!dest.exists()); + } +} diff --git a/src-tauri/src/commands.rs b/src-tauri/src/commands.rs index d07334a..72466a4 100644 --- a/src-tauri/src/commands.rs +++ b/src-tauri/src/commands.rs @@ -20,6 +20,9 @@ use crate::append::{self, AppendCache, AppendInput, AppendOptions, AppendPreview use crate::archive::{self, ArchiveCache, ZipEntryInfo}; use crate::clipboard::{self, CopyFormat}; use crate::cluster::{self, ClusterCache, ClusterReport, ClusterSpec}; +use crate::columnar_export::{ + ColumnarExportOptions, ColumnarExportReport, ColumnarExportReportCache, +}; use crate::compare::{self, CompareCache, CompareInfo, ComparePage, CompareSpec, DiffStatus}; use crate::crossval::{self, CrossRule, CrossValCache, CrossValReport}; use crate::dedup::{self, DedupCache, DedupSpec, DuplicateKeepStrategy, DuplicateReport}; @@ -51,6 +54,7 @@ use crate::json_import::{JsonImportOptions, JsonImportPreview, JsonImportPreview use crate::outlier::{ self, CachedOutlier, OutlierAction, OutlierActionPreview, OutlierCache, OutlierSpec, }; +use crate::parquet_arrow::{ColumnarInspection, ColumnarOpenOptions}; use crate::parse::{parse, ParseSettings, ParsedFile}; use crate::paste::{self, PasteOptions, PastePreview}; use crate::pii::{self, CachedPii, PiiCache, PiiSpec, RedactionAction, RedactionPreview}; @@ -498,7 +502,17 @@ pub async fn start_convert_to_editable( return Err(AppError::invalid("document is already editable")); } if !force { - if let Some(path) = doc.path.as_deref() { + // F32 columnar documents carry their own open-time estimate (the + // CSV sampler cannot read a binary parquet/arrow file). + if let Some(columnar) = doc.columnar_handle() { + if columnar.convert_needs_decision() { + return Err(AppError::invalid(format!( + "the estimated in-memory size is about {} MB, which may exhaust memory — \ + export a slice instead, or convert anyway", + columnar.editable_estimate() / (1024 * 1024) + ))); + } + } else if let Some(path) = doc.path.as_deref() { let est = index::estimate(path)?; if est.needs_decision { return Err(AppError::invalid(format!( @@ -517,26 +531,41 @@ pub async fn start_convert_to_editable( let _ = crate::job::run_blocking(ctx, move |ctx| { // Stream the rows out under a read lock (reads stay available), // then commit under a brief write lock, revision-guarded. - let (rows, revision) = { + let (rows, schemas, revision) = { let doc = handle.read().map_err(poisoned)?; if doc.is_editable() { return Err(AppError::invalid("document is already editable")); } - let n = doc.n_rows(); - ctx.set_total(n as u64); - let mut rows: Vec> = Vec::with_capacity(n); - let mut pending = 0u64; - doc.visit_rows(0..n, &mut |_, row| { - rows.push(row.to_vec()); - pending += 1; - if pending >= 4096 { - ctx.advance(pending)?; - pending = 0; + // F32 columnar documents convert through the Option plane so + // NULL stays distinct from the empty string: each + // null-containing column gets a collision-free null token + // recorded in its schema. + if let Some(columnar) = doc.columnar_handle() { + let plan = crate::parquet_arrow::plan_editable(columnar, Some(ctx))?; + let mut schemas = plan.schemas; + for (i, schema) in schemas.iter_mut().enumerate() { + if let Some(id) = doc.column_ids().get(i) { + schema.column_id = id.clone(); + } } - Ok(true) - })?; - ctx.advance(pending)?; - (rows, doc.revision()) + (plan.rows, Some(schemas), doc.revision()) + } else { + let n = doc.n_rows(); + ctx.set_total(n as u64); + let mut rows: Vec> = Vec::with_capacity(n); + let mut pending = 0u64; + doc.visit_rows(0..n, &mut |_, row| { + rows.push(row.to_vec()); + pending += 1; + if pending >= 4096 { + ctx.advance(pending)?; + pending = 0; + } + Ok(true) + })?; + ctx.advance(pending)?; + (rows, None, doc.revision()) + } }; ctx.check()?; // last cancellation point before the commit @@ -545,6 +574,19 @@ pub async fn start_convert_to_editable( // materialised rows would no longer match. doc.check_revision(revision)?; doc.make_editable(rows)?; + if let Some(schemas) = schemas { + for schema in schemas { + doc.set_column_schema(schema); + } + // F32: the editable rows no longer mirror the binary + // parquet/arrow file — cut the path so a subsequent Save can + // never write CSV bytes over the columnar source. The + // document continues as an unsaved derived table (Save asks + // for a new destination; closing warns). + doc.path = None; + doc.set_fingerprint(None); + doc.mark_derived_unsaved(); + } Ok(()) }) .await; @@ -570,6 +612,13 @@ pub async fn start_reindex( "only indexed documents reload by re-indexing", )); } + // The CSV indexer cannot read a binary columnar file; F32 documents + // reload by reopening the file through the columnar open flow. + if doc.columnar_handle().is_some() { + return Err(AppError::invalid( + "Parquet/Arrow documents reload by reopening the file", + )); + } let path = doc .path .clone() @@ -4787,3 +4836,201 @@ pub async fn json_export( }); Ok(job_id) } + +// ----- Parquet / Arrow interop (F32) ------------------------------------------- + +/// Inspect a Parquet / Arrow IPC file BEFORE any open: container format +/// (Feather v2 IS the Arrow IPC file format — say so in UI copy), row and +/// row-group/batch counts, columns mapped to the F31 logical types, +/// compression codecs, nested (complex) fields, and the editable-memory +/// estimate with its decision flag. +#[tauri::command] +pub async fn columnar_inspect(path: String) -> AppResult { + tauri::async_runtime::spawn_blocking(move || { + crate::parquet_arrow::inspect(Path::new(&path), None) + }) + .await + .map_err(|e| AppError::Other(format!("background task failed: {e}")))? +} + +/// Open a Parquet / Arrow file as an indexed READ-ONLY document: windowed +/// columnar reads behind the same grid/filter/export machinery as an F10 +/// indexed CSV, with convert-to-editable available later +/// (`start_convert_to_editable` handles columnar documents). Runs under the +/// F10 job kind ("openIndexed") so the front end's existing completion path +/// adds the tab; the document registers under the returned doc id when the +/// job finishes. Explode policies are rejected here — they change the row +/// count, which an indexed backing cannot represent; use +/// `columnar_open_editable`. Creates no on-disk caches, so cancellation +/// leaves nothing behind. +#[tauri::command] +pub async fn columnar_open_indexed( + path: String, + options: Option, + app: tauri::AppHandle, + state: Db<'_>, + jobs: State<'_, JobRegistry>, +) -> AppResult { + let options = options.unwrap_or_default(); + let doc_id = lock(&state)?.alloc_id(); + let ctx = jobs.begin_for_app(&app, "openIndexed", Some(doc_id)); + let job_id = ctx.id; + let app_for_job = app.clone(); + tauri::async_runtime::spawn(async move { + let _ = crate::job::run_blocking(ctx, move |ctx| { + use tauri::Manager; + let source = PathBuf::from(&path); + let fingerprint = util::stat_fingerprint(&source); + let file = crate::parquet_arrow::open_indexed(&source, &options, Some(ctx))?; + let mut doc = Document::from_columnar(doc_id, Some(source), file); + doc.set_fingerprint(fingerprint); + let registry = app_for_job.state::>(); + registry + .lock() + .map_err(|_| AppError::Other("internal state lock error".into()))? + .insert(doc); + Ok(()) + }) + .await; + }); + Ok(IndexedOpenStart { job_id, doc_id }) +} + +/// Open a Parquet / Arrow file straight into a fully editable in-memory +/// document, honouring EVERY complex-field policy including exploding one +/// list column into rows. Re-runs the memory estimate first; pass `force` +/// after an explicit user decision. NULLs stay distinct from empty strings +/// via collision-free per-column null tokens recorded on the generated +/// schemas. The document opens UNSAVED with no path: its rows no longer +/// mirror the binary columnar file, and Save must never write CSV bytes +/// over a .parquet/.arrow source. +#[tauri::command] +pub async fn columnar_open_editable( + path: String, + options: Option, + force: bool, + app: tauri::AppHandle, + state: Db<'_>, + jobs: State<'_, JobRegistry>, +) -> AppResult { + let options = options.unwrap_or_default(); + if !force { + let probe = path.clone(); + let inspection = tauri::async_runtime::spawn_blocking(move || { + crate::parquet_arrow::inspect(Path::new(&probe), None) + }) + .await + .map_err(|e| AppError::Other(format!("background task failed: {e}")))??; + if inspection.needs_decision { + return Err(AppError::invalid(format!( + "the estimated in-memory size is about {} MB, which may exhaust memory — \ + open read-only (indexed) instead, or open editable anyway", + inspection.estimated_memory / (1024 * 1024) + ))); + } + } + let doc_id = lock(&state)?.alloc_id(); + let ctx = jobs.begin_for_app(&app, "openIndexed", Some(doc_id)); + let job_id = ctx.id; + let app_for_job = app.clone(); + tauri::async_runtime::spawn(async move { + let _ = crate::job::run_blocking(ctx, move |ctx| { + use tauri::Manager; + let table = + crate::parquet_arrow::open_editable_rows(Path::new(&path), &options, Some(ctx))?; + let n_cols = table.headers.len(); + let mut records = Vec::with_capacity(table.rows.len() + 1); + records.push(table.headers); + records.extend(table.rows); + let parsed = ParsedFile { + records, + n_cols, + delimiter: b',', + encoding: encoding_rs::UTF_8, + had_bom: false, + uses_crlf: false, + import: crate::parse::ImportInfo::default(), + }; + let mut doc = Document::from_parsed(doc_id, None, parsed, true); + for (i, mut schema) in table.schemas.into_iter().enumerate() { + if let Some(id) = doc.column_ids().get(i) { + schema.column_id = id.clone(); + } + doc.set_column_schema(schema); + } + doc.mark_derived_unsaved(); + let registry = app_for_job.state::>(); + registry + .lock() + .map_err(|_| AppError::Other("internal state lock error".into()))? + .insert(doc); + Ok(()) + }) + .await; + }); + Ok(IndexedOpenStart { job_id, doc_id }) +} + +/// Start a scoped, typed export to Parquet (uncompressed/Snappy/Zstd) or +/// Arrow IPC file/stream as a cancellable job (kind "export"). Typing rules, +/// null semantics and the invalid-cell policy (null + counted warning) are +/// documented on [`crate::columnar_export`]. The scope resolves and the +/// revision is checked BEFORE the job spawns (and re-checked inside it); +/// everything streams through the atomic-save pipeline, so failure or +/// cancellation removes the staging file and never touches an existing +/// destination. Fetch the outcome (rows, bytes, per-column warning counts) +/// with `get_columnar_export_report` after the `job-finished` event. +#[allow(clippy::too_many_arguments)] +#[tauri::command] +pub async fn columnar_export( + doc_id: u64, + path: String, + options: ColumnarExportOptions, + scope: ExportScope, + expected_revision: u64, + app: tauri::AppHandle, + state: Db<'_>, + jobs: State<'_, JobRegistry>, + reports: State<'_, ColumnarExportReportCache>, +) -> AppResult { + let handle = doc_handle(&state, doc_id)?; + { + // Fail fast: a stale snapshot or an invalid scope rejects the + // invoke, not a background job. + let doc = handle.read().map_err(poisoned)?; + doc.check_revision(expected_revision)?; + export_scope::resolve_scope(&doc, &scope)?; + } + + let sink = reports.share(); + let ctx = jobs.begin_for_app(&app, "export", Some(doc_id)); + let job_id = ctx.id; + tauri::async_runtime::spawn(async move { + let _ = crate::job::run_blocking(ctx, move |ctx| { + let doc = handle.read().map_err(poisoned)?; + let report = crate::columnar_export::run( + &doc, + Path::new(&path), + &options, + &scope, + expected_revision, + ctx, + )?; + if let Ok(mut map) = sink.lock() { + map.insert(job_id, report); + } + Ok(()) + }) + .await; + }); + Ok(job_id) +} + +/// The report of a finished columnar export, by its job id. +#[tauri::command] +pub fn get_columnar_export_report( + job_id: u64, + reports: State<'_, ColumnarExportReportCache>, +) -> Option { + reports.get(job_id) +} diff --git a/src-tauri/src/document.rs b/src-tauri/src/document.rs index 347d62d..de9479b 100644 --- a/src-tauri/src/document.rs +++ b/src-tauri/src/document.rs @@ -248,13 +248,18 @@ impl SummaryAccumulator { } } -/// How a document's rows are stored (F10). +/// How a document's rows are stored (F10, F32). pub enum Backing { /// Fully materialised and mutable (the default). Memory, /// Streaming, read-only access through a record index; `rows` stays /// empty and every mutation fails with [`AppError::ReadOnly`]. Indexed(IndexHandle), + /// Read-only windowed access over a columnar file (F32: Parquet / Arrow + /// IPC). Like `Indexed`, `rows` stays empty and mutations fail; the text + /// plane renders columnar NULL as an empty cell while the handle keeps + /// the null-vs-empty distinction for export and conversion. + Columnar(crate::parquet_arrow::ColumnarHandle), } /// An open document. @@ -543,6 +548,68 @@ impl Document { } } + /// Build a read-only document over an open columnar file (F32: + /// Parquet / Arrow IPC). Headers are the flattened path-based column + /// names (always real names — never synthetic), and the generated F31 + /// schemas are attached keyed by the positional column IDs. + pub fn from_columnar( + id: u64, + path: Option, + columnar: crate::parquet_arrow::ColumnarFile, + ) -> Document { + let crate::parquet_arrow::ColumnarFile { + handle, + headers, + schemas, + } = columnar; + let n_cols = headers.len(); + let column_ids = positional_column_ids(n_cols); + let mut schema = crate::schema::DocumentSchema::default(); + for (i, mut col_schema) in schemas.into_iter().enumerate() { + col_schema.column_id = column_ids[i].clone(); + schema.set_column(col_schema); + } + Document { + id, + path, + headers, + rows: Vec::new(), + has_header_row: true, + delimiter: b',', + encoding_name: "UTF-8".to_string(), + had_bom: false, + line_ending: LineEnding::Lf, + dirty_cells: HashSet::new(), + undo_stack: Vec::new(), + redo_stack: Vec::new(), + undo_meta: Vec::new(), + redo_meta: Vec::new(), + next_op_id: 0, + journal: None, + follow: false, + follow_range_from: None, + saved_marker: 0, + column_ids, + next_column_id: n_cols as u64, + filter_rows: None, + view_sort: Vec::new(), + filter_view: None, + revision: 1, + col_revisions: vec![1; n_cols], + filter_revision: 1, + import_info: ImportInfo::default(), + fingerprint: None, + schema, + schema_revision: 0, + schema_issues: Vec::new(), + dictionary: crate::dictionary::Dictionary::default(), + dictionary_revision: 0, + backing: Backing::Columnar(handle), + archive: None, + archive_guard: None, + } + } + // ----- accessors ------------------------------------------------------- pub fn n_cols(&self) -> usize { @@ -553,6 +620,7 @@ impl Document { match &self.backing { Backing::Memory => self.rows.len(), Backing::Indexed(handle) => handle.n_data_records(), + Backing::Columnar(handle) => handle.n_rows(), } } @@ -770,11 +838,41 @@ impl Document { self.follow_range_from = from; } - /// Wire name of the backing, carried on [`DocumentMeta`]. + /// Wire name of the backing, carried on [`DocumentMeta`]. Columnar (F32) + /// documents report `indexedReadOnly` DELIBERATELY: every front-end + /// affordance for the F10 indexed mode (read-only gating, the + /// convert-to-editable flow) applies to them unchanged. pub fn backing_name(&self) -> &'static str { match self.backing { Backing::Memory => "editable", - Backing::Indexed(_) => "indexedReadOnly", + Backing::Indexed(_) | Backing::Columnar(_) => "indexedReadOnly", + } + } + + /// The columnar handle behind an F32 Parquet/Arrow document, when this + /// document has one. Export and conversion use it to read the `Option` + /// plane (columnar NULL = `None`, empty string = `Some("")`), which the + /// rectangular text plane cannot carry. + pub fn columnar_handle(&self) -> Option<&crate::parquet_arrow::ColumnarHandle> { + match &self.backing { + Backing::Columnar(handle) => Some(handle), + _ => None, + } + } + + /// F32: the absolute row ranges a filter scan must visit, when the + /// columnar backing's row-group statistics prove the skipped groups + /// cannot match `spec`. `None` = scan everything (non-columnar backings, + /// no usable statistics, or no eligible conditions). + pub fn filter_scan_ranges( + &self, + spec: &crate::dto::FilterGroup, + ) -> Option>> { + match &self.backing { + Backing::Columnar(handle) => { + handle.filter_scan_ranges(spec, &|col| self.column_schema_at(col).cloned()) + } + _ => None, } } @@ -816,6 +914,7 @@ impl Document { Ok(()) } Backing::Indexed(handle) => handle.visit(range, f), + Backing::Columnar(handle) => handle.visit(range, f), } } @@ -839,6 +938,7 @@ impl Document { Ok(()) } Backing::Indexed(handle) => handle.visit_at(indices, f), + Backing::Columnar(handle) => handle.visit_at(indices, f), } } @@ -1236,7 +1336,7 @@ impl Document { fn summary_scan_len(&self) -> usize { match self.backing { Backing::Memory => self.n_rows(), - Backing::Indexed(_) => self.n_rows().min(INDEXED_SUMMARY_SAMPLE), + Backing::Indexed(_) | Backing::Columnar(_) => self.n_rows().min(INDEXED_SUMMARY_SAMPLE), } } diff --git a/src-tauri/src/filter.rs b/src-tauri/src/filter.rs index 17c1c0a..174ad3f 100644 --- a/src-tauri/src/filter.rs +++ b/src-tauri/src/filter.rs @@ -66,16 +66,27 @@ fn norm(s: &str, case_sensitive: bool) -> String { /// Evaluate a filter spec over every data row, returning matching absolute /// row indices in document order. Streams through [`Document::visit_rows`], -/// so it works for both editable and indexed backings. +/// so it works for every backing. On a columnar (F32 Parquet) document, +/// row-group statistics can prove entire row groups unmatchable for the +/// required equality/range conditions; those groups are skipped — the +/// pruning is conservative, so the result is IDENTICAL to the full scan. pub fn matching_rows(doc: &Document, spec: &FilterGroup) -> AppResult> { let compiled = compile_group(spec, &|col| doc.column_schema_at(col).cloned())?; let mut out = Vec::new(); - doc.visit_rows(0..doc.n_rows(), &mut |i, row| { + let mut visit = |i: usize, row: &[String]| { if eval(&compiled, row) { out.push(i); } Ok(true) - })?; + }; + match doc.filter_scan_ranges(spec) { + Some(ranges) => { + for range in ranges { + doc.visit_rows(range, &mut visit)?; + } + } + None => doc.visit_rows(0..doc.n_rows(), &mut visit)?, + } Ok(out) } diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index 3035b66..f9ad838 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -11,6 +11,11 @@ mod append; mod archive; mod clipboard; mod cluster; +/// F32 Parquet/Arrow write side: scoped typed exports (declared F31 logical +/// types → arrow types, invalid cells → null + counted warning) to Parquet +/// and Arrow IPC through the atomic-save pipeline. Public like +/// [`parquet_arrow`] so the test harness exercises the round trip directly. +pub mod columnar_export; mod commands; mod compare; mod crossval; @@ -55,6 +60,12 @@ mod json_export; /// JSON export stage. pub mod json_import; mod outlier; +/// Public like [`job`]: the F32 Parquet/Arrow read engine (format detection, +/// inspection, the indexed read-only columnar backing with typed-text +/// conversion and row-group statistics pruning, nested-field policies, and +/// editable materialisation) consumed by the F32 command surface, the export +/// stage and the test harness. +pub mod parquet_arrow; mod parse; mod paste; mod pii; @@ -164,6 +175,7 @@ pub fn run() { .manage(crate::pii::PiiCache::default()) .manage(crate::follow::FollowRegistry::default()) .manage(crate::json_import::JsonImportPreviewCache::default()) + .manage(crate::columnar_export::ColumnarExportReportCache::default()) .manage(crate::project::ProjectStore::default()) .manage(crate::annotations::AnnotationRegistry::default()) .manage(crate::highlight::HighlightStore::default()) @@ -261,6 +273,11 @@ pub fn run() { commands::get_json_import_preview, commands::json_import_apply, commands::json_export, + commands::columnar_inspect, + commands::columnar_open_indexed, + commands::columnar_open_editable, + commands::columnar_export, + commands::get_columnar_export_report, commands::get_settings, commands::set_settings, commands::validate_profile, diff --git a/src-tauri/src/parquet_arrow.rs b/src-tauri/src/parquet_arrow.rs new file mode 100644 index 0000000..6bba37c --- /dev/null +++ b/src-tauri/src/parquet_arrow.rs @@ -0,0 +1,3282 @@ +//! Parquet & Arrow interop (F32) — read side. +//! +//! Opens typed columnar datasets — Apache Parquet, Arrow IPC files and Arrow +//! IPC streams (Feather v2 IS the Arrow IPC file format; UI copy should say +//! so) — preserving types and nulls: +//! +//! - **Inspection** ([`inspect`]): row count, columns mapped to F31 +//! [`LogicalType`]s, row-group/batch count, compression codecs, nested +//! fields, and a rough estimate of what the fully editable in-memory +//! document would cost. +//! - **Indexed read-only backing** ([`open_indexed`] → [`ColumnarHandle`], +//! wired into [`crate::document::Backing::Columnar`]): windowed reads over +//! parquet row groups / Arrow record batches with a bounded LRU of decoded +//! text blocks, so the grid, filters and export all work through the same +//! `visit_rows` machinery as the F10 CSV index. +//! - **Typed-value → text conversion**: the text plane is CANONICAL under the +//! generated F31 column schemas, so classified cells round-trip exactly — +//! integers (i64/u64 well beyond JS number range) as plain decimal strings, +//! decimals rendered exactly from mantissa+scale (`1.50` keeps its scale), +//! floats via Rust's shortest round-trip display, booleans as +//! `true`/`false`, dates as `%Y-%m-%d`, naive timestamps as ISO wall time +//! (with an `inputFormats` pattern carrying fractional seconds), zoned +//! timestamps as the UTC instant (`...Z`) with the original zone kept in +//! [`ColumnSchema::time_zone`], and binary as lowercase hex. +//! - **Null vs empty string**: a columnar NULL is `None` in the +//! [`TabularSource`] contract; an empty string is `Some("")`. The +//! read-only text plane (`visit`/`visit_at`, what the grid shows) renders +//! NULL as an empty cell; the `Option` plane keeps the distinction for +//! export and conversion. [`plan_editable`] / [`open_editable_rows`] +//! preserve it in editable documents by assigning each null-containing +//! column a collision-free null token (`NULL`, escalating to `NULL#1`, …, +//! never colliding with the column's actual trimmed values) recorded in the +//! column's schema, so `is_null_token` recovers the null bit later. +//! - **Nested data**: structs are ALWAYS flattened to path-based names +//! (segments escaped by [`crate::json_import::escape_key`], joined with +//! `.`); lists and maps follow an explicit [`ComplexPolicy`] — preserve as +//! JSON text (deterministic; note serde_json object keys serialize sorted), +//! reject (drop) the field, or explode a SINGLE list column into rows +//! (editable open only — an indexed backing cannot re-number rows). +//! Null and empty lists both yield one row with a null cell under explode; +//! under preserve-as-JSON they stay distinct (`None` vs `[]`). +//! - **Row-group statistics pruning** ([`ColumnarHandle::filter_scan_ranges`], +//! used by [`crate::filter::matching_rows`]): equality and range conditions +//! on numeric/date/datetime columns skip parquet row groups whose min/max +//! statistics prove no row can match. Pruning is CONSERVATIVE: it only ever +//! skips a group when the typed bounds make a match impossible under the +//! column's CURRENT schema (which must still agree with the open-time +//! schema on every parse-relevant field), so filtered and unfiltered reads +//! return identical values for matching rows. Everything else falls back to +//! the full scan. +//! +//! Deliberate deferrals (documented so later stages don't guess): write-side +//! export (backend part 2); stats pruning for unsigned 32/64-bit columns +//! (parquet stores them sign-reinterpreted, the old-format ordering is +//! unreliable) and Decimal256; exploding maps or more than one list column; +//! and JSON-policy object key order (serde_json sorts keys). +//! +//! Cancellation: open/inspect/convert observe [`JobCtx`] cooperatively. The +//! read side creates NO on-disk caches (the block LRU is in memory), so a +//! cancelled open or convert leaves nothing behind by construction. + +use std::collections::BTreeMap; +use std::collections::HashMap; +use std::fs::File; +use std::io::{BufReader, Read}; +use std::ops::Range; +use std::path::{Path, PathBuf}; +use std::sync::Mutex; + +use arrow::array::{ + Array, ArrayRef, BinaryArray, BinaryViewArray, BooleanArray, Date32Array, Date64Array, + Decimal128Array, Decimal256Array, FixedSizeBinaryArray, FixedSizeListArray, Float16Array, + Float32Array, Float64Array, Int16Array, Int32Array, Int64Array, Int8Array, LargeBinaryArray, + LargeListArray, LargeStringArray, ListArray, MapArray, StringArray, StringViewArray, + StructArray, Time32MillisecondArray, Time32SecondArray, Time64MicrosecondArray, + Time64NanosecondArray, TimestampMicrosecondArray, TimestampMillisecondArray, + TimestampNanosecondArray, TimestampSecondArray, UInt16Array, UInt32Array, UInt64Array, + UInt8Array, +}; +use arrow::datatypes::{DataType, Schema as ArrowSchema, TimeUnit}; +use arrow::ipc::reader::{FileReader as IpcFileReader, StreamReader as IpcStreamReader}; +use arrow::record_batch::RecordBatch; +use chrono::{DateTime, NaiveDateTime, NaiveTime}; +use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; +use parquet::file::metadata::ParquetMetaData; +use parquet::file::statistics::Statistics; +use serde::{Deserialize, Serialize}; + +use crate::dto::{ + Conjunction, FileFingerprint, FilterCondition, FilterGroup, FilterNode, FilterOp, +}; +use crate::error::{AppError, AppResult}; +use crate::job::JobCtx; +use crate::json_import::escape_key; +use crate::schema::{ + compare_typed, parse_typed, ColumnSchema, DecimalValue, LogicalType, TypedValue, +}; +use crate::tabular::{ContentFingerprint, RowCountHint, TabularColumn, TabularRow, TabularSource}; +use crate::{index, util}; + +/// Rows per decoded text block (matches the F10 index visit block, so one +/// grid window is at most two block decodes). +const BLOCK_ROWS: usize = 4096; + +/// Default in-memory budget for the decoded-block LRU. Bounded regardless of +/// file size; a single oversized block is always retained (the cache never +/// thrashes itself below one block). +const DEFAULT_CACHE_BUDGET: usize = 32 * 1024 * 1024; + +/// Per-cell / per-row overhead constants shared with the open-time memory +/// estimate (mirrors [`index::estimate`]'s deliberately rough model). +const CELL_OVERHEAD: u64 = 40; +const ROW_OVERHEAD: u64 = 32; + +fn arrow_err(e: impl std::fmt::Display) -> AppError { + AppError::Other(format!("columnar read error: {e}")) +} + +// --------------------------------------------------------------------------- +// Format detection +// --------------------------------------------------------------------------- + +/// The three supported columnar container formats. Deserializable so the +/// F32 export options ([`crate::columnar_export`]) name their target with +/// the same wire values the inspection reports. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum ColumnarFormat { + Parquet, + /// Arrow IPC file (a.k.a. Feather v2 — same container, keep the alias in + /// UI copy). + ArrowFile, + ArrowStream, +} + +impl ColumnarFormat { + pub fn wire_name(self) -> &'static str { + match self { + ColumnarFormat::Parquet => "parquet", + ColumnarFormat::ArrowFile => "arrowFile", + ColumnarFormat::ArrowStream => "arrowStream", + } + } +} + +/// Sniff the container format: `PAR1` / `ARROW1` magic first, then an Arrow +/// IPC stream probe (streams have no magic; validity is the probe). +pub fn detect_format(path: &Path) -> AppResult { + let mut head = [0u8; 8]; + let n = { + let mut file = File::open(path)?; + let mut read = 0usize; + loop { + let got = file.read(&mut head[read..])?; + if got == 0 { + break; + } + read += got; + if read == head.len() { + break; + } + } + read + }; + if n >= 4 && &head[..4] == b"PAR1" { + return Ok(ColumnarFormat::Parquet); + } + if n >= 6 && &head[..6] == b"ARROW1" { + return Ok(ColumnarFormat::ArrowFile); + } + match IpcStreamReader::try_new(BufReader::new(File::open(path)?), None) { + Ok(_) => Ok(ColumnarFormat::ArrowStream), + Err(_) => Err(AppError::invalid( + "not a recognised columnar file (expected Parquet, an Arrow IPC file, or an Arrow IPC stream)", + )), + } +} + +// --------------------------------------------------------------------------- +// Open options / nested-field policies +// --------------------------------------------------------------------------- + +/// What to do with a field the text contract cannot carry directly (list, +/// map, and other non-flattenable nested types). +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum ComplexPolicy { + /// Keep the field as one column of canonical JSON text (F31 `json` type). + #[default] + PreserveJson, + /// Multiply the record into one row per list element (editable open + /// only, one list column per open). + Explode, + /// Drop the field from the projected schema. + Reject, +} + +/// Options for [`inspect`] / [`open_indexed`] / [`open_editable_rows`] +/// (wire DTO, camelCase). +#[derive(Debug, Clone, Default, Deserialize)] +#[serde(rename_all = "camelCase", default)] +pub struct ColumnarOpenOptions { + /// Default policy for complex fields. + pub complex_policy: ComplexPolicy, + /// Per-field overrides, keyed by the flattened path-based column name. + pub field_policies: BTreeMap, + /// Override for the decoded-block LRU budget (bytes). `0` = default. + pub cache_budget_bytes: usize, +} + +// --------------------------------------------------------------------------- +// Projection: arrow schema -> output columns +// --------------------------------------------------------------------------- + +/// How one output column reads its value. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum OutKind { + /// A primitive leaf, rendered to canonical text. + Primitive, + /// A complex field kept as canonical JSON text. + Json, + /// A list column exploded into rows (editable open only). + Explode, +} + +/// One projected output column. +#[derive(Debug, Clone)] +struct OutCol { + /// Flattened path-based name (struct segments escaped and dot-joined). + name: String, + kind: OutKind, + /// Descent from the record batch: `steps[0]` is the top-level column, + /// the rest are struct child indices. + steps: Vec, + /// The leaf's dictionary-resolved data type (rendering target). + data_type: DataType, + logical: LogicalType, + time_zone: Option, + input_formats: Option>, + nullable: bool, + /// Index of the column's FIRST parquet leaf, for row-group statistics. + leaf_start: usize, +} + +struct Projection { + cols: Vec, + /// Whether the parquet-leaf accounting is trustworthy (no exotic types + /// encountered that could desynchronise leaf indices). + stats_ok: bool, + /// Total parquet leaves consumed by the FULL arrow schema (including + /// rejected fields), for cross-checking against file metadata. + total_leaves: usize, +} + +/// Whether `dt` is text-representable as a single canonical cell. +fn is_primitive(dt: &DataType) -> bool { + match dt { + DataType::Null + | DataType::Boolean + | DataType::Int8 + | DataType::Int16 + | DataType::Int32 + | DataType::Int64 + | DataType::UInt8 + | DataType::UInt16 + | DataType::UInt32 + | DataType::UInt64 + | DataType::Float16 + | DataType::Float32 + | DataType::Float64 + | DataType::Decimal128(_, _) + | DataType::Decimal256(_, _) + | DataType::Utf8 + | DataType::LargeUtf8 + | DataType::Utf8View + | DataType::Binary + | DataType::LargeBinary + | DataType::BinaryView + | DataType::FixedSizeBinary(_) + | DataType::Date32 + | DataType::Date64 + | DataType::Time32(_) + | DataType::Time64(_) + | DataType::Timestamp(_, _) + | DataType::Duration(_) + | DataType::Interval(_) => true, + DataType::Dictionary(_, value) => is_primitive(value), + _ => false, + } +} + +/// Strip dictionary wrappers to the rendering target type. +fn resolved_type(dt: &DataType) -> DataType { + match dt { + DataType::Dictionary(_, value) => resolved_type(value), + other => other.clone(), + } +} + +/// F31 logical type + schema metadata for a primitive leaf type. +fn logical_of(dt: &DataType) -> (LogicalType, Option, Option>) { + match dt { + DataType::Boolean => (LogicalType::Boolean, None, None), + DataType::Int8 + | DataType::Int16 + | DataType::Int32 + | DataType::Int64 + | DataType::UInt8 + | DataType::UInt16 + | DataType::UInt32 + | DataType::UInt64 => (LogicalType::Integer, None, None), + DataType::Float16 | DataType::Float32 | DataType::Float64 => { + (LogicalType::Float, None, None) + } + DataType::Decimal128(_, _) | DataType::Decimal256(_, _) => { + (LogicalType::Decimal, None, None) + } + DataType::Date32 | DataType::Date64 => (LogicalType::Date, None, None), + DataType::Timestamp(_, tz) => match tz { + // Zoned: rendered as the UTC instant (RFC 3339 `...Z`, covered by + // the built-in parse fallback); the original zone is metadata. + Some(tz) => (LogicalType::Datetime, Some(tz.to_string()), None), + // Naive wall time: the custom pattern accepts an optional + // fractional part, so sub-second units round-trip. + None => ( + LogicalType::Datetime, + None, + Some(vec!["%Y-%m-%dT%H:%M:%S%.f".to_string()]), + ), + }, + DataType::Utf8 | DataType::LargeUtf8 | DataType::Utf8View => { + (LogicalType::Text, None, None) + } + // Times, durations, intervals, binary (hex) and Null all land as text. + _ => (LogicalType::Text, None, None), + } +} + +/// Number of parquet leaf columns a field of this type occupies (depth-first, +/// matching the parquet schema layout arrow writers produce). +fn leaf_count(dt: &DataType) -> usize { + match dt { + DataType::Struct(fields) => fields.iter().map(|f| leaf_count(f.data_type())).sum(), + DataType::List(f) | DataType::LargeList(f) | DataType::FixedSizeList(f, _) => { + leaf_count(f.data_type()) + } + DataType::Map(entries, _) => leaf_count(entries.data_type()), + DataType::Dictionary(_, value) => leaf_count(value), + DataType::RunEndEncoded(_, values) => leaf_count(values.data_type()), + DataType::Union(fields, _) => fields.iter().map(|(_, f)| leaf_count(f.data_type())).sum(), + _ => 1, + } +} + +/// Types whose presence anywhere in the schema makes the depth-first leaf +/// accounting unreliable; statistics pruning is disabled wholesale then. +fn leaf_accounting_fragile(dt: &DataType) -> bool { + matches!( + dt, + DataType::Null | DataType::Union(_, _) | DataType::RunEndEncoded(_, _) + ) +} + +struct ProjectionBuilder<'a> { + options: &'a ColumnarOpenOptions, + allow_explode: bool, + cols: Vec, + stats_ok: bool, + leaf_cursor: usize, + explode_seen: bool, +} + +impl ProjectionBuilder<'_> { + fn policy_for(&self, path: &str) -> ComplexPolicy { + self.options + .field_policies + .get(path) + .copied() + .unwrap_or(self.options.complex_policy) + } + + fn walk( + &mut self, + field: &arrow::datatypes::Field, + prefix: Option<&str>, + steps: &mut Vec, + nullable_so_far: bool, + ) -> AppResult<()> { + let name = match prefix { + Some(p) => format!("{p}.{}", escape_key(field.name())), + None => escape_key(field.name()), + }; + let nullable = nullable_so_far || field.is_nullable(); + let dt = field.data_type(); + if leaf_accounting_fragile(dt) { + self.stats_ok = false; + } + if let DataType::Struct(children) = dt { + // Structs are ALWAYS flattened: stable path-based names. + for (i, child) in children.iter().enumerate() { + steps.push(i); + self.walk(child, Some(&name), steps, nullable)?; + steps.pop(); + } + return Ok(()); + } + if is_primitive(dt) { + let resolved = resolved_type(dt); + if leaf_accounting_fragile(&resolved) { + self.stats_ok = false; + } + let (logical, time_zone, input_formats) = logical_of(&resolved); + self.cols.push(OutCol { + name, + kind: OutKind::Primitive, + steps: steps.clone(), + data_type: resolved, + logical, + time_zone, + input_formats, + nullable, + leaf_start: self.leaf_cursor, + }); + self.leaf_cursor += leaf_count(dt); + return Ok(()); + } + // Complex field: list / map / other — explicit policy. + let policy = self.policy_for(&name); + match policy { + ComplexPolicy::Reject => {} + ComplexPolicy::PreserveJson => { + self.cols.push(OutCol { + name, + kind: OutKind::Json, + steps: steps.clone(), + data_type: resolved_type(dt), + logical: LogicalType::Json, + time_zone: None, + input_formats: None, + nullable, + leaf_start: self.leaf_cursor, + }); + } + ComplexPolicy::Explode => { + if !self.allow_explode { + return Err(AppError::invalid(format!( + "field \"{name}\": explode changes the row count, which an indexed \ + read-only document cannot represent — open it as editable instead, \ + or choose preserveJson/reject" + ))); + } + let element = match dt { + DataType::List(f) | DataType::LargeList(f) | DataType::FixedSizeList(f, _) => f, + _ => { + return Err(AppError::invalid(format!( + "field \"{name}\": only list columns can be exploded into rows \ + (maps and other nested types support preserveJson or reject)" + ))) + } + }; + if self.explode_seen { + return Err(AppError::invalid( + "only one list column can be exploded per open; choose preserveJson or \ + reject for the others", + )); + } + self.explode_seen = true; + let elem_type = resolved_type(element.data_type()); + let (logical, time_zone, input_formats) = if is_primitive(&elem_type) { + logical_of(&elem_type) + } else { + (LogicalType::Json, None, None) + }; + self.cols.push(OutCol { + name, + kind: OutKind::Explode, + steps: steps.clone(), + data_type: resolved_type(dt), + logical, + time_zone, + input_formats, + nullable: true, + leaf_start: self.leaf_cursor, + }); + } + } + self.leaf_cursor += leaf_count(dt); + Ok(()) + } +} + +fn build_projection( + schema: &ArrowSchema, + options: &ColumnarOpenOptions, + allow_explode: bool, +) -> AppResult { + let mut builder = ProjectionBuilder { + options, + allow_explode, + cols: Vec::new(), + stats_ok: true, + leaf_cursor: 0, + explode_seen: false, + }; + let mut steps = Vec::new(); + for (i, field) in schema.fields().iter().enumerate() { + steps.push(i); + builder.walk(field, None, &mut steps, false)?; + steps.pop(); + } + if builder.cols.is_empty() { + return Err(AppError::invalid( + "no readable columns (every field was rejected or the schema is empty)", + )); + } + Ok(Projection { + cols: builder.cols, + stats_ok: builder.stats_ok, + total_leaves: builder.leaf_cursor, + }) +} + +/// The generated F31 schema for one output column. Column IDs are positional +/// placeholders (`c{i}`), matching what [`crate::document::Document`] assigns. +fn column_schemas(cols: &[OutCol]) -> Vec { + cols.iter() + .enumerate() + .map(|(i, col)| { + let mut s = ColumnSchema::new(format!("c{i}"), col.name.clone(), col.logical); + s.nullable = col.nullable; + s.time_zone = col.time_zone.clone(); + s.input_formats = col.input_formats.clone(); + s + }) + .collect() +} + +// --------------------------------------------------------------------------- +// Typed value -> canonical text +// --------------------------------------------------------------------------- + +/// Exact decimal text from a mantissa's decimal digits and a scale. +fn decimal_text_from_digits(negative: bool, digits: &str, scale: i32) -> String { + let digits = digits.trim_start_matches('0'); + let mut digits = if digits.is_empty() { "0" } else { digits }.to_string(); + let negative = negative && digits != "0"; + if scale <= 0 { + if digits != "0" { + digits.push_str(&"0".repeat((-scale) as usize)); + } + DecimalValue { + negative, + digits, + scale: 0, + } + .to_plain_string() + } else { + DecimalValue { + negative, + digits, + scale: scale as u32, + } + .to_plain_string() + } +} + +fn decimal128_text(v: i128, scale: i32) -> String { + let negative = v < 0; + let digits = v.unsigned_abs().to_string(); + decimal_text_from_digits(negative, &digits, scale) +} + +/// Exact [`DecimalValue`] from an i128 mantissa (for statistics pruning). +fn decimal128_value(v: i128, scale: i32) -> DecimalValue { + let text = decimal128_text(v, scale); + let negative = text.starts_with('-'); + let unsigned = text.trim_start_matches('-'); + let (int_part, frac_part) = match unsigned.split_once('.') { + Some((i, f)) => (i, f), + None => (unsigned, ""), + }; + let combined = format!("{int_part}{frac_part}"); + let trimmed = combined.trim_start_matches('0'); + DecimalValue { + negative: negative && !trimmed.is_empty(), + digits: if trimmed.is_empty() { + "0".to_string() + } else { + trimmed.to_string() + }, + scale: frac_part.len() as u32, + } +} + +fn hex_text(bytes: &[u8]) -> String { + let mut out = String::with_capacity(bytes.len() * 2); + for b in bytes { + out.push_str(&format!("{b:02x}")); + } + out +} + +fn timestamp_naive(v: i64, unit: &TimeUnit) -> Option { + match unit { + TimeUnit::Second => DateTime::from_timestamp(v, 0), + TimeUnit::Millisecond => DateTime::from_timestamp_millis(v), + TimeUnit::Microsecond => DateTime::from_timestamp_micros(v), + TimeUnit::Nanosecond => Some(DateTime::from_timestamp_nanos(v)), + } + .map(|dt| dt.naive_utc()) +} + +/// ISO text for a timestamp; zoned columns carry the UTC instant with an +/// explicit `Z`. Values outside chrono's ±262143-year range fall back to the +/// raw tick count (reads stay total; such values are astronomically rare). +fn timestamp_text(v: i64, unit: &TimeUnit, zoned: bool) -> String { + match timestamp_naive(v, unit) { + Some(ndt) => { + if zoned { + ndt.format("%Y-%m-%dT%H:%M:%S%.fZ").to_string() + } else { + ndt.format("%Y-%m-%dT%H:%M:%S%.f").to_string() + } + } + None => v.to_string(), + } +} + +fn date32_text(days: i32) -> String { + match DateTime::from_timestamp(i64::from(days) * 86_400, 0) { + Some(dt) => dt.date_naive().format("%Y-%m-%d").to_string(), + None => days.to_string(), + } +} + +fn date64_text(ms: i64) -> String { + match DateTime::from_timestamp_millis(ms) { + Some(dt) => dt.date_naive().format("%Y-%m-%d").to_string(), + None => ms.to_string(), + } +} + +fn time_text(secs: i64, nanos: u32) -> String { + match u32::try_from(secs) + .ok() + .and_then(|s| NaiveTime::from_num_seconds_from_midnight_opt(s, nanos)) + { + Some(t) => t.format("%H:%M:%S%.f").to_string(), + None => format!("{secs}.{nanos:09}"), + } +} + +macro_rules! int_arm { + ($arr:expr, $i:expr, $ty:ty) => {{ + let a = $arr + .as_any() + .downcast_ref::<$ty>() + .ok_or_else(|| arrow_err("array type mismatch"))?; + a.value($i).to_string() + }}; +} + +/// Render one primitive cell to canonical text (`None` = columnar NULL). +fn render_primitive(arr: &dyn Array, i: usize) -> AppResult> { + if arr.is_null(i) { + return Ok(None); + } + let text = match arr.data_type() { + DataType::Null => return Ok(None), + DataType::Boolean => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + a.value(i).to_string() + } + DataType::Int8 => int_arm!(arr, i, Int8Array), + DataType::Int16 => int_arm!(arr, i, Int16Array), + DataType::Int32 => int_arm!(arr, i, Int32Array), + DataType::Int64 => int_arm!(arr, i, Int64Array), + DataType::UInt8 => int_arm!(arr, i, UInt8Array), + DataType::UInt16 => int_arm!(arr, i, UInt16Array), + DataType::UInt32 => int_arm!(arr, i, UInt32Array), + DataType::UInt64 => int_arm!(arr, i, UInt64Array), + DataType::Float16 => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + a.value(i).to_f32().to_string() + } + DataType::Float32 => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + a.value(i).to_string() + } + DataType::Float64 => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + a.value(i).to_string() + } + DataType::Decimal128(_, scale) => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + decimal128_text(a.value(i), i32::from(*scale)) + } + DataType::Decimal256(_, scale) => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + let s = a.value(i).to_string(); + let negative = s.starts_with('-'); + decimal_text_from_digits(negative, s.trim_start_matches('-'), i32::from(*scale)) + } + DataType::Utf8 => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + a.value(i).to_string() + } + DataType::LargeUtf8 => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + a.value(i).to_string() + } + DataType::Utf8View => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + a.value(i).to_string() + } + DataType::Binary => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + hex_text(a.value(i)) + } + DataType::LargeBinary => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + hex_text(a.value(i)) + } + DataType::BinaryView => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + hex_text(a.value(i)) + } + DataType::FixedSizeBinary(_) => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + hex_text(a.value(i)) + } + DataType::Date32 => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + date32_text(a.value(i)) + } + DataType::Date64 => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + date64_text(a.value(i)) + } + DataType::Time32(TimeUnit::Second) => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + time_text(i64::from(a.value(i)), 0) + } + DataType::Time32(TimeUnit::Millisecond) => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + let v = i64::from(a.value(i)); + time_text( + v.div_euclid(1_000), + (v.rem_euclid(1_000) * 1_000_000) as u32, + ) + } + DataType::Time64(TimeUnit::Microsecond) => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + let v = a.value(i); + time_text( + v.div_euclid(1_000_000), + (v.rem_euclid(1_000_000) * 1_000) as u32, + ) + } + DataType::Time64(TimeUnit::Nanosecond) => { + let a = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + let v = a.value(i); + time_text( + v.div_euclid(1_000_000_000), + v.rem_euclid(1_000_000_000) as u32, + ) + } + DataType::Timestamp(unit, tz) => { + let v = timestamp_ticks(arr, i, unit)?; + timestamp_text(v, unit, tz.is_some()) + } + DataType::Dictionary(_, value) => { + let cast = arrow::compute::cast(arr, value).map_err(arrow_err)?; + return render_primitive(cast.as_ref(), i); + } + // Durations, intervals and anything else exotic: arrow's own display + // (deterministic, documented as non-round-tripping text). + _ => { + let options = arrow::util::display::FormatOptions::default(); + let fmt = + arrow::util::display::ArrayFormatter::try_new(arr, &options).map_err(arrow_err)?; + fmt.value(i).try_to_string().map_err(arrow_err)? + } + }; + Ok(Some(text)) +} + +fn timestamp_ticks(arr: &dyn Array, i: usize, unit: &TimeUnit) -> AppResult { + Ok(match unit { + TimeUnit::Second => arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))? + .value(i), + TimeUnit::Millisecond => arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))? + .value(i), + TimeUnit::Microsecond => arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))? + .value(i), + TimeUnit::Nanosecond => arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))? + .value(i), + }) +} + +// --------------------------------------------------------------------------- +// Complex values -> canonical JSON +// --------------------------------------------------------------------------- + +/// One value as a `serde_json::Value`. Exact where JSON is exact (i64/u64 +/// carry full precision as JSON numbers; decimals, timestamps and binary +/// become strings so nothing is rounded). +fn json_at(arr: &dyn Array, i: usize) -> AppResult { + use serde_json::Value; + if arr.is_null(i) { + return Ok(Value::Null); + } + Ok(match arr.data_type() { + DataType::Boolean => Value::Bool( + arr.as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))? + .value(i), + ), + DataType::Int8 | DataType::Int16 | DataType::Int32 | DataType::Int64 => { + let text = render_primitive(arr, i)?.unwrap_or_default(); + let v: i64 = text.parse().map_err(arrow_err)?; + Value::Number(v.into()) + } + DataType::UInt8 | DataType::UInt16 | DataType::UInt32 | DataType::UInt64 => { + let text = render_primitive(arr, i)?.unwrap_or_default(); + let v: u64 = text.parse().map_err(arrow_err)?; + Value::Number(v.into()) + } + DataType::Float16 | DataType::Float32 | DataType::Float64 => { + let text = render_primitive(arr, i)?.unwrap_or_default(); + let v: f64 = text.parse().map_err(arrow_err)?; + match serde_json::Number::from_f64(v) { + Some(n) => Value::Number(n), + None => Value::String(text), + } + } + DataType::Struct(fields) => { + let s = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + let mut obj = serde_json::Map::new(); + for (field, column) in fields.iter().zip(s.columns()) { + obj.insert(field.name().clone(), json_at(column.as_ref(), i)?); + } + Value::Object(obj) + } + DataType::List(_) => { + let l = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + json_list(&l.value(i))? + } + DataType::LargeList(_) => { + let l = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + json_list(&l.value(i))? + } + DataType::FixedSizeList(_, _) => { + let l = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + json_list(&l.value(i))? + } + DataType::Map(_, _) => { + let m = arr + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))?; + let entries = m.value(i); + let keys = entries.column(0); + let values = entries.column(1); + let mut obj = serde_json::Map::new(); + for e in 0..entries.len() { + let key = render_primitive(keys.as_ref(), e)?.unwrap_or_else(|| "null".to_string()); + obj.insert(key, json_at(values.as_ref(), e)?); + } + Value::Object(obj) + } + DataType::Dictionary(_, value) => { + let cast = arrow::compute::cast(arr, value).map_err(arrow_err)?; + json_at(cast.as_ref(), i)? + } + // Every remaining primitive renders through its canonical text + // (decimals, dates, timestamps, times, binary-as-hex, strings). + _ => match render_primitive(arr, i)? { + Some(text) => Value::String(text), + None => Value::Null, + }, + }) +} + +fn json_list(elems: &ArrayRef) -> AppResult { + let mut out = Vec::with_capacity(elems.len()); + for e in 0..elems.len() { + out.push(json_at(elems.as_ref(), e)?); + } + Ok(serde_json::Value::Array(out)) +} + +// --------------------------------------------------------------------------- +// Batch -> text rows +// --------------------------------------------------------------------------- + +/// Per-batch reader state for one output column: the structs along the +/// descent (for ancestor-null checks) and the leaf array. +struct LeafCursor { + parents: Vec, + leaf: ArrayRef, +} + +impl LeafCursor { + fn new(col: &OutCol, batch: &RecordBatch) -> AppResult { + let mut parents = Vec::new(); + let mut current: ArrayRef = batch + .columns() + .get(col.steps[0]) + .ok_or_else(|| arrow_err("record batch is narrower than the schema"))? + .clone(); + for &step in &col.steps[1..] { + let s = current + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("schema mismatch: expected a struct"))?; + let child = s + .columns() + .get(step) + .ok_or_else(|| arrow_err("schema mismatch: struct child out of range"))? + .clone(); + parents.push(current); + current = child; + } + // Dictionary-encoded primitives decode once per batch, not per cell. + if col.kind == OutKind::Primitive + && matches!(current.data_type(), DataType::Dictionary(_, _)) + { + current = arrow::compute::cast(current.as_ref(), &col.data_type).map_err(arrow_err)?; + } + Ok(LeafCursor { + parents, + leaf: current, + }) + } + + /// A cell is null when the leaf OR any ancestor struct is null. + fn cell(&self, col: &OutCol, row: usize) -> AppResult> { + for parent in &self.parents { + if parent.is_null(row) { + return Ok(None); + } + } + match col.kind { + OutKind::Primitive => render_primitive(self.leaf.as_ref(), row), + OutKind::Json => { + if self.leaf.is_null(row) { + Ok(None) + } else { + let value = json_at(self.leaf.as_ref(), row)?; + Ok(Some(serde_json::to_string(&value).map_err(|e| { + arrow_err(format!("JSON encoding failed: {e}")) + })?)) + } + } + OutKind::Explode => Err(AppError::Other( + "internal: explode columns are handled by the editable open".into(), + )), + } + } +} + +/// Append every row of `batch` to `out` as optional text cells. +fn render_batch(cols: &[OutCol], batch: &RecordBatch, out: &mut Vec) -> AppResult<()> { + let cursors: Vec = cols + .iter() + .map(|c| LeafCursor::new(c, batch)) + .collect::>()?; + for row in 0..batch.num_rows() { + let mut cells = Vec::with_capacity(cols.len()); + for (col, cursor) in cols.iter().zip(&cursors) { + cells.push(cursor.cell(col, row)?); + } + out.push(cells); + } + Ok(()) +} + +// --------------------------------------------------------------------------- +// Row-group statistics (parquet pruning) +// --------------------------------------------------------------------------- + +/// Typed min/max of one output column within one row group. +#[derive(Debug, Clone)] +struct StatsRange { + min: TypedValue, + max: TypedValue, +} + +/// Sign-extend big-endian two's-complement bytes into an i128 (decimal +/// statistics are stored this way for byte-array physical types). +fn i128_from_be(bytes: &[u8]) -> Option { + if bytes.is_empty() || bytes.len() > 16 { + return None; + } + let fill = if bytes[0] & 0x80 != 0 { 0xFF } else { 0x00 }; + let mut buf = [fill; 16]; + buf[16 - bytes.len()..].copy_from_slice(bytes); + Some(i128::from_be_bytes(buf)) +} + +/// Widen an f32-derived bound so text-plane parsing (shortest decimal of the +/// f32, re-parsed as f64) can never fall outside the pruning interval. +fn widen_down(v: f64) -> f64 { + v - v.abs() * 1e-6 - f64::MIN_POSITIVE +} +fn widen_up(v: f64) -> f64 { + v + v.abs() * 1e-6 + f64::MIN_POSITIVE +} + +/// Extract the typed min/max for `col` from one row group, when the +/// statistics exist and can be trusted for the column's logical type. +fn stat_range(col: &OutCol, stats: &Statistics) -> Option { + let range = |min: TypedValue, max: TypedValue| Some(StatsRange { min, max }); + match (&col.data_type, stats) { + // Signed integers (and the two small unsigned widths whose values are + // non-negative in a signed physical column, so ordering agrees). + ( + DataType::Int8 | DataType::Int16 | DataType::Int32 | DataType::UInt8 | DataType::UInt16, + Statistics::Int32(s), + ) => { + let (min, max) = (*s.min_opt()?, *s.max_opt()?); + range( + TypedValue::Integer(i128::from(min)), + TypedValue::Integer(i128::from(max)), + ) + } + (DataType::Int64, Statistics::Int64(s)) => { + let (min, max) = (*s.min_opt()?, *s.max_opt()?); + range( + TypedValue::Integer(i128::from(min)), + TypedValue::Integer(i128::from(max)), + ) + } + // UInt32/UInt64 are stored sign-reinterpreted with unsigned sort + // order; older writers ordered them signed. Deliberately skipped. + (DataType::Float32, Statistics::Float(s)) => { + let (min, max) = (*s.min_opt()?, *s.max_opt()?); + if min.is_nan() || max.is_nan() { + return None; + } + range( + TypedValue::Float(widen_down(f64::from(min))), + TypedValue::Float(widen_up(f64::from(max))), + ) + } + (DataType::Float64, Statistics::Double(s)) => { + let (min, max) = (*s.min_opt()?, *s.max_opt()?); + if min.is_nan() || max.is_nan() { + return None; + } + range(TypedValue::Float(min), TypedValue::Float(max)) + } + (DataType::Decimal128(_, scale), Statistics::Int32(s)) => { + let (min, max) = (*s.min_opt()?, *s.max_opt()?); + range( + TypedValue::Decimal(decimal128_value(i128::from(min), i32::from(*scale))), + TypedValue::Decimal(decimal128_value(i128::from(max), i32::from(*scale))), + ) + } + (DataType::Decimal128(_, scale), Statistics::Int64(s)) => { + let (min, max) = (*s.min_opt()?, *s.max_opt()?); + range( + TypedValue::Decimal(decimal128_value(i128::from(min), i32::from(*scale))), + TypedValue::Decimal(decimal128_value(i128::from(max), i32::from(*scale))), + ) + } + (DataType::Decimal128(_, scale), Statistics::FixedLenByteArray(s)) => { + // Byte-array decimal ordering was wrong in the deprecated stats + // fields; only trust the modern min_value/max_value pair. + if stats.is_min_max_deprecated() { + return None; + } + let min = i128_from_be(s.min_opt()?.data())?; + let max = i128_from_be(s.max_opt()?.data())?; + range( + TypedValue::Decimal(decimal128_value(min, i32::from(*scale))), + TypedValue::Decimal(decimal128_value(max, i32::from(*scale))), + ) + } + (DataType::Decimal128(_, scale), Statistics::ByteArray(s)) => { + if stats.is_min_max_deprecated() { + return None; + } + let min = i128_from_be(s.min_opt()?.data())?; + let max = i128_from_be(s.max_opt()?.data())?; + range( + TypedValue::Decimal(decimal128_value(min, i32::from(*scale))), + TypedValue::Decimal(decimal128_value(max, i32::from(*scale))), + ) + } + (DataType::Date32, Statistics::Int32(s)) => { + let (min, max) = (*s.min_opt()?, *s.max_opt()?); + let to_date = |days: i32| { + DateTime::from_timestamp(i64::from(days) * 86_400, 0).map(|d| d.date_naive()) + }; + range( + TypedValue::Date(to_date(min)?), + TypedValue::Date(to_date(max)?), + ) + } + (DataType::Timestamp(unit, _), Statistics::Int64(s)) => { + let (min, max) = (*s.min_opt()?, *s.max_opt()?); + range( + TypedValue::DateTime(timestamp_naive(min, unit)?), + TypedValue::DateTime(timestamp_naive(max, unit)?), + ) + } + _ => None, + } +} + +/// Per-row-group, per-output-column stats, when the leaf accounting checks +/// out against the file's actual parquet schema. +fn extract_stats( + meta: &ParquetMetaData, + cols: &[OutCol], + projection: &Projection, +) -> Option>>> { + if !projection.stats_ok { + return None; + } + if meta.file_metadata().schema_descr().num_columns() != projection.total_leaves { + return None; + } + let mut out = Vec::with_capacity(meta.num_row_groups()); + for rg in meta.row_groups() { + let mut per_col = Vec::with_capacity(cols.len()); + for col in cols { + let entry = if col.kind == OutKind::Primitive { + rg.columns() + .get(col.leaf_start) + .and_then(|c| c.statistics()) + .and_then(|s| stat_range(col, s)) + } else { + None + }; + per_col.push(entry); + } + out.push(per_col); + } + Some(out) +} + +// --------------------------------------------------------------------------- +// The columnar handle (indexed read-only backing) +// --------------------------------------------------------------------------- + +#[derive(Debug)] +struct CachedBlock { + rows: Vec, + bytes: usize, + stamp: u64, +} + +#[derive(Debug)] +struct BlockCache { + blocks: HashMap, + bytes: usize, + budget: usize, + next_stamp: u64, +} + +impl BlockCache { + fn new(budget: usize) -> BlockCache { + BlockCache { + blocks: HashMap::new(), + bytes: 0, + budget, + next_stamp: 0, + } + } + + fn touch(&mut self, block: usize) -> bool { + self.next_stamp += 1; + let stamp = self.next_stamp; + match self.blocks.get_mut(&block) { + Some(b) => { + b.stamp = stamp; + true + } + None => false, + } + } + + fn insert(&mut self, block: usize, rows: Vec) { + let bytes = block_bytes(&rows); + self.next_stamp += 1; + let stamp = self.next_stamp; + if let Some(old) = self + .blocks + .insert(block, CachedBlock { rows, bytes, stamp }) + { + self.bytes -= old.bytes; + } + self.bytes += bytes; + // Evict least-recently-used blocks past the budget; the newest block + // always survives so an oversized block still works. + while self.bytes > self.budget && self.blocks.len() > 1 { + let Some((&victim, _)) = self.blocks.iter().min_by_key(|(_, b)| b.stamp) else { + break; + }; + if victim == block { + break; + } + if let Some(old) = self.blocks.remove(&victim) { + self.bytes -= old.bytes; + } + } + } +} + +fn block_bytes(rows: &[TabularRow]) -> usize { + let mut total = 0usize; + for row in rows { + total += ROW_OVERHEAD as usize; + for cell in row { + total += CELL_OVERHEAD as usize + cell.as_ref().map_or(0, |s| s.len()); + } + } + total +} + +/// Read-only handle over a columnar file: windowed text reads through a +/// bounded LRU of decoded blocks. The document integration mirrors +/// [`index::IndexHandle`] (`visit` / `visit_at` over `&[String]`); the +/// `Option` plane ([`ColumnarHandle::read_optional`]) additionally keeps +/// columnar NULL (`None`) distinct from the empty string (`Some("")`). +#[derive(Debug)] +pub struct ColumnarHandle { + path: PathBuf, + format: ColumnarFormat, + cols: Vec, + schemas: Vec, + /// Prefix sums of the per-chunk (row group / record batch) row counts + /// (len = chunks + 1; last = row count). + chunk_starts: Vec, + n_rows: usize, + /// Identity of the source file as of the open; every read re-validates + /// it so an in-place rewrite errors instead of decoding stale bytes. + source_check: Option, + editable_estimate: u64, + /// Parquet only: per-row-group typed min/max per output column. + stats: Option>>>, + cache: Mutex, +} + +impl ColumnarHandle { + pub fn n_rows(&self) -> usize { + self.n_rows + } + + pub fn n_cols(&self) -> usize { + self.cols.len() + } + + pub fn format(&self) -> ColumnarFormat { + self.format + } + + /// Flattened output column names, in read order. + pub fn headers(&self) -> Vec { + self.cols.iter().map(|c| c.name.clone()).collect() + } + + /// Generated F31 schemas, parallel to [`ColumnarHandle::headers`] + /// (column IDs are positional `c{i}` placeholders). + pub fn schemas(&self) -> &[ColumnSchema] { + &self.schemas + } + + /// Rough bytes a fully editable in-memory conversion would need. + pub fn editable_estimate(&self) -> u64 { + self.editable_estimate + } + + /// Whether converting to editable should require explicit confirmation. + pub fn convert_needs_decision(&self) -> bool { + self.editable_estimate > index::MEMORY_DECISION_THRESHOLD + } + + #[cfg(test)] + fn cached_bytes(&self) -> usize { + self.cache.lock().map(|c| c.bytes).unwrap_or(0) + } + + #[cfg(test)] + fn cached_blocks(&self) -> usize { + self.cache.lock().map(|c| c.blocks.len()).unwrap_or(0) + } + + /// Tabular columns for the [`TabularSource`] adapter and export stages. + pub fn tabular_columns(&self) -> Vec { + self.cols + .iter() + .zip(&self.schemas) + .enumerate() + .map(|(i, (col, schema))| TabularColumn { + name: col.name.clone(), + id: Some(format!("c{i}")), + schema: Some(schema.clone()), + }) + .collect() + } + + // ----- reads ----------------------------------------------------------- + + fn check_source(&self) -> AppResult<()> { + if let Some(expected) = self.source_check { + if util::stat_fingerprint(&self.path) != Some(expected) { + return Err(AppError::Other( + "the source file changed on disk; reload the document".into(), + )); + } + } + Ok(()) + } + + /// Run `f` over the cached rows of `block`, decoding it on a miss. The + /// cache lock is held for the duration of `f`; callers must NOT invoke + /// user callbacks inside it (they copy what they need out instead), so a + /// callback that re-enters this handle cannot deadlock. + fn with_block(&self, block: usize, f: impl FnOnce(&[TabularRow]) -> T) -> AppResult { + let mut cache = self + .cache + .lock() + .map_err(|_| AppError::Other("columnar cache lock poisoned".into()))?; + if !cache.touch(block) { + let rows = self.decode_block(block)?; + cache.insert(block, rows); + } + let cached = cache + .blocks + .get(&block) + .expect("block inserted or touched above"); + Ok(f(&cached.rows)) + } + + /// Decode text rows for block `block` (rows `[block*BLOCK_ROWS, ...)`). + fn decode_block(&self, block: usize) -> AppResult> { + self.check_source()?; + let start = block * BLOCK_ROWS; + let end = (start + BLOCK_ROWS).min(self.n_rows); + if start >= end { + return Ok(Vec::new()); + } + let mut rows = Vec::with_capacity(end - start); + match self.format { + ColumnarFormat::Parquet => { + let file = File::open(&self.path)?; + let builder = ParquetRecordBatchReaderBuilder::try_new(file).map_err(arrow_err)?; + let reader = builder + .with_batch_size(BLOCK_ROWS.min(end - start)) + .with_offset(start) + .with_limit(end - start) + .build() + .map_err(arrow_err)?; + for batch in reader { + let batch = batch.map_err(arrow_err)?; + render_batch(&self.cols, &batch, &mut rows)?; + } + } + ColumnarFormat::ArrowFile => { + let mut reader = + IpcFileReader::try_new(BufReader::new(File::open(&self.path)?), None) + .map_err(arrow_err)?; + let first = self.chunk_of(start); + let last = self.chunk_of(end - 1); + for chunk in first..=last { + reader.set_index(chunk).map_err(arrow_err)?; + let batch = reader + .next() + .ok_or_else(|| arrow_err("record batch missing (file truncated?)"))? + .map_err(arrow_err)?; + self.render_chunk_slice(&batch, chunk, start, end, &mut rows)?; + } + } + ColumnarFormat::ArrowStream => { + let reader = + IpcStreamReader::try_new(BufReader::new(File::open(&self.path)?), None) + .map_err(arrow_err)?; + let first = self.chunk_of(start); + let last = self.chunk_of(end - 1); + for (chunk, batch) in reader.enumerate() { + if chunk > last { + break; + } + let batch = batch.map_err(arrow_err)?; + if chunk < first { + continue; + } + self.render_chunk_slice(&batch, chunk, start, end, &mut rows)?; + } + } + } + if rows.len() != end - start { + return Err(AppError::Other( + "the source file changed on disk; reload the document".into(), + )); + } + Ok(rows) + } + + fn render_chunk_slice( + &self, + batch: &RecordBatch, + chunk: usize, + start: usize, + end: usize, + rows: &mut Vec, + ) -> AppResult<()> { + let chunk_start = self.chunk_starts[chunk]; + let chunk_end = self.chunk_starts[chunk + 1]; + if batch.num_rows() != chunk_end - chunk_start { + return Err(AppError::Other( + "the source file changed on disk; reload the document".into(), + )); + } + let lo = start.max(chunk_start) - chunk_start; + let hi = end.min(chunk_end) - chunk_start; + let sliced = batch.slice(lo, hi - lo); + render_batch(&self.cols, &sliced, rows) + } + + /// Which chunk (row group / batch) contains absolute row `row`. + fn chunk_of(&self, row: usize) -> usize { + debug_assert!(row < self.n_rows); + match self.chunk_starts.binary_search(&row) { + Ok(c) => c, + Err(insert) => insert - 1, + } + } + + /// Owned `Option`-plane rows `[offset, offset+limit)` — columnar NULL is + /// `None`, an empty string is `Some("")`. Window semantics mirror + /// [`TabularSource::read_rows`]. + pub fn read_optional( + &self, + offset: u64, + limit: usize, + ctx: Option<&JobCtx>, + ) -> AppResult> { + if let Some(ctx) = ctx { + ctx.check()?; + } + if limit == 0 { + return Ok(Vec::new()); + } + let start = usize::try_from(offset) + .unwrap_or(usize::MAX) + .min(self.n_rows); + let end = start.saturating_add(limit).min(self.n_rows); + let mut out = Vec::with_capacity(end - start); + let mut at = start; + while at < end { + if let Some(ctx) = ctx { + ctx.check()?; + } + let block = at / BLOCK_ROWS; + let block_start = block * BLOCK_ROWS; + let hi = end.min(block_start + BLOCK_ROWS); + let mut copied = self.with_block(block, |rows| { + rows[at - block_start..hi - block_start].to_vec() + })?; + out.append(&mut copied); + at = hi; + } + Ok(out) + } + + /// Owned TEXT-plane rows for `[lo, hi)` within one block (NULL renders as + /// an empty cell). Copies out under the cache lock so callbacks never run + /// while it is held. + fn text_rows(&self, block: usize, lo: usize, hi: usize) -> AppResult>> { + self.with_block(block, |rows| { + rows[lo..hi] + .iter() + .map(|row| { + row.iter() + .map(|c| c.clone().unwrap_or_default()) + .collect::>() + }) + .collect() + }) + } + + /// Visit data rows `[range)` in order over the TEXT plane (NULL renders + /// as an empty cell). Mirrors [`index::IndexHandle::visit`]. + pub fn visit( + &self, + range: Range, + f: &mut dyn FnMut(usize, &[String]) -> AppResult, + ) -> AppResult<()> { + let end = range.end.min(self.n_rows); + let mut start = range.start.min(end); + while start < end { + let block = start / BLOCK_ROWS; + let block_start = block * BLOCK_ROWS; + let hi = end.min(block_start + BLOCK_ROWS); + let rows = self.text_rows(block, start - block_start, hi - block_start)?; + for (i, row) in rows.iter().enumerate() { + if !f(start + i, row)? { + return Ok(()); + } + } + start = hi; + } + Ok(()) + } + + /// Visit specific rows in CALLER order (text plane). Mirrors + /// [`index::IndexHandle::visit_at`]. + pub fn visit_at( + &self, + indices: &[usize], + f: &mut dyn FnMut(usize, &[String]) -> AppResult, + ) -> AppResult<()> { + if let Some(&bad) = indices.iter().find(|&&i| i >= self.n_rows) { + return Err(AppError::invalid(format!("row {bad} is out of range"))); + } + for &i in indices { + let block = i / BLOCK_ROWS; + let block_start = block * BLOCK_ROWS; + let row = self + .text_rows(block, i - block_start, i - block_start + 1)? + .pop() + .expect("one row requested"); + if !f(i, &row)? { + return Ok(()); + } + } + Ok(()) + } + + // ----- statistics pruning ------------------------------------------------ + + /// Absolute row ranges a filter scan must visit, when row-group + /// statistics prove the remaining groups cannot match. `None` = no + /// pruning possible (scan everything); pruning is conservative, so rows + /// outside the returned ranges are GUARANTEED not to match `spec`. + /// + /// `schema_of` must resolve the document's CURRENT per-column schemas — + /// the same ones [`crate::filter`] compiles conditions against. + pub fn filter_scan_ranges( + &self, + spec: &FilterGroup, + schema_of: &dyn Fn(usize) -> Option, + ) -> Option>> { + let stats = self.stats.as_ref()?; + let mut conds = Vec::new(); + required_conditions(spec, &mut conds); + if conds.is_empty() { + return None; + } + let mut tests = Vec::new(); + for c in conds { + if let Some(test) = self.prune_test(c, schema_of) { + tests.push(test); + } + } + if tests.is_empty() { + return None; + } + let mut ranges: Vec> = Vec::new(); + for (group, per_col) in stats.iter().enumerate() { + let skip = tests.iter().any(|t| { + per_col + .get(t.column) + .and_then(|r| r.as_ref()) + .is_some_and(|r| t.impossible(r)) + }); + if skip { + continue; + } + let start = self.chunk_starts[group]; + let end = self.chunk_starts[group + 1]; + match ranges.last_mut() { + Some(last) if last.end == start => last.end = end, + _ => ranges.push(start..end), + } + } + Some(ranges) + } + + /// Compile one condition into a pruning test, when it is eligible: an + /// equality or range comparison, on a numeric/date/datetime column whose + /// current schema still parses exactly like the open-time one. + fn prune_test( + &self, + c: &FilterCondition, + schema_of: &dyn Fn(usize) -> Option, + ) -> Option { + let op = match c.op { + FilterOp::Equals => PruneOp::Eq, + FilterOp::Gt => PruneOp::Gt, + FilterOp::Gte => PruneOp::Gte, + FilterOp::Lt => PruneOp::Lt, + FilterOp::Lte => PruneOp::Lte, + _ => return None, + }; + let opened = self.schemas.get(c.column)?; + let current = schema_of(c.column)?; + if !(current.logical_type.is_numeric() || current.logical_type.is_temporal()) { + return None; + } + // The statistics describe text produced under the OPEN-TIME schema; + // if any parse-relevant field moved since, classification could + // disagree with the stats domain — fall back to the full scan. + if current.logical_type != opened.logical_type + || current.locale != opened.locale + || current.time_zone != opened.time_zone + || current.input_formats != opened.input_formats + { + return None; + } + let trimmed = c.value.trim(); + if trimmed.is_empty() { + return None; + } + // Text equality compares the FULL cell text; canonical numeric text + // never carries surrounding whitespace, so a padded value would need + // the full scan (it simply never matches — but stay conservative). + if op == PruneOp::Eq && trimmed != c.value { + return None; + } + let target = parse_typed(trimmed, ¤t).ok()?; + Some(PruneTest { + column: c.column, + op, + target, + }) + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PruneOp { + Eq, + Gt, + Gte, + Lt, + Lte, +} + +struct PruneTest { + column: usize, + op: PruneOp, + target: TypedValue, +} + +impl PruneTest { + /// Whether the row group's [min, max] makes any match impossible. + fn impossible(&self, range: &StatsRange) -> bool { + use std::cmp::Ordering; + // compare_typed falls back to Equal across variants, which would + // corrupt range pruning — require the exact same variant. + if std::mem::discriminant(&self.target) != std::mem::discriminant(&range.min) { + return false; + } + let min = compare_typed(&range.min, &self.target); + let max = compare_typed(&range.max, &self.target); + match self.op { + PruneOp::Eq => min == Ordering::Greater || max == Ordering::Less, + PruneOp::Gt => max != Ordering::Greater, + PruneOp::Gte => max == Ordering::Less, + PruneOp::Lt => min != Ordering::Less, + PruneOp::Lte => min == Ordering::Greater, + } + } +} + +/// Conditions that are REQUIRED for a row to match: every condition reachable +/// from the root through `And` groups (an `Or` group with a single node is +/// equivalent to `And`). OR branches contribute nothing (conservative). +fn required_conditions<'a>(group: &'a FilterGroup, out: &mut Vec<&'a FilterCondition>) { + if group.conjunction == Conjunction::Or && group.nodes.len() != 1 { + return; + } + for node in &group.nodes { + match node { + FilterNode::Condition(c) => out.push(c), + FilterNode::Group(sub) => required_conditions(sub, out), + } + } +} + +// --------------------------------------------------------------------------- +// Inspection +// --------------------------------------------------------------------------- + +/// One column as reported by [`inspect`] (wire DTO, camelCase). +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct InspectedColumn { + /// Flattened path-based name. + pub name: String, + /// The arrow type, for display (`Int64`, `Timestamp(µs, Europe/Berlin)`…). + pub arrow_type: String, + /// The F31 logical type the column maps to. + pub logical_type: LogicalType, + pub nullable: bool, + #[serde(skip_serializing_if = "Option::is_none")] + pub time_zone: Option, + /// Whether this column came out of a nested field (struct flattening or + /// a complex-field policy). + pub nested: bool, +} + +/// Everything the open dialog needs BEFORE opening (wire DTO, camelCase). +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ColumnarInspection { + pub format: String, + pub row_count: u64, + /// Parquet row groups / Arrow record batches. + pub chunk_count: u64, + /// Distinct parquet compression codecs, `None` for Arrow IPC. + #[serde(skip_serializing_if = "Option::is_none")] + pub compression: Option, + /// Columns under the DEFAULT policies (complex fields preserved as JSON). + pub columns: Vec, + /// Flattened paths of complex fields that take a [`ComplexPolicy`]. + pub complex_fields: Vec, + /// Rough bytes the fully editable in-memory document would need. + pub estimated_memory: u64, + /// Whether opening editable should require an explicit choice. + pub needs_decision: bool, + pub file_size: u64, +} + +fn arrow_type_label(field_type: &DataType) -> String { + format!("{field_type}") +} + +fn estimate_memory(n_rows: u64, n_cols: u64, data_bytes: u64) -> u64 { + data_bytes + n_rows * n_cols * CELL_OVERHEAD + n_rows * ROW_OVERHEAD +} + +/// Shared open-time scan: schema + per-chunk row counts (+ parquet metadata). +struct Scanned { + schema: ArrowSchema, + chunk_rows: Vec, + data_bytes: u64, + compression: Option, + parquet_meta: Option>, +} + +fn scan_source(path: &Path, format: ColumnarFormat, ctx: Option<&JobCtx>) -> AppResult { + if let Some(ctx) = ctx { + ctx.check()?; + } + match format { + ColumnarFormat::Parquet => { + let file = File::open(path)?; + let builder = ParquetRecordBatchReaderBuilder::try_new(file).map_err(arrow_err)?; + let schema = builder.schema().as_ref().clone(); + let meta = builder.metadata().clone(); + let mut chunk_rows = Vec::with_capacity(meta.num_row_groups()); + let mut data_bytes = 0u64; + let mut codecs: Vec = Vec::new(); + for rg in meta.row_groups() { + if let Some(ctx) = ctx { + ctx.check()?; + } + chunk_rows.push(usize::try_from(rg.num_rows()).unwrap_or(0)); + data_bytes = data_bytes.saturating_add(rg.total_byte_size().max(0) as u64); + for col in rg.columns() { + let name = format!("{}", col.compression()); + if !codecs.contains(&name) { + codecs.push(name); + } + } + } + Ok(Scanned { + schema, + chunk_rows, + data_bytes, + compression: if codecs.is_empty() { + None + } else { + Some(codecs.join(", ")) + }, + parquet_meta: Some(meta), + }) + } + ColumnarFormat::ArrowFile | ColumnarFormat::ArrowStream => { + // One pass over the batches for per-chunk row counts, decoding a + // single projected column to keep it cheap. + let file_size = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0); + let (schema, chunk_rows) = if format == ColumnarFormat::ArrowFile { + let full = IpcFileReader::try_new(BufReader::new(File::open(path)?), None) + .map_err(arrow_err)?; + let schema = full.schema().as_ref().clone(); + drop(full); + let reader = + IpcFileReader::try_new(BufReader::new(File::open(path)?), Some(vec![0])) + .map_err(arrow_err)?; + let mut rows = Vec::new(); + for batch in reader { + if let Some(ctx) = ctx { + ctx.check()?; + } + rows.push(batch.map_err(arrow_err)?.num_rows()); + } + (schema, rows) + } else { + let reader = + IpcStreamReader::try_new(BufReader::new(File::open(path)?), Some(vec![0])) + .map_err(arrow_err)?; + let schema = reader.schema().as_ref().clone(); + let mut rows = Vec::new(); + for batch in reader { + if let Some(ctx) = ctx { + ctx.check()?; + } + rows.push(batch.map_err(arrow_err)?.num_rows()); + } + (schema, rows) + }; + if schema.fields().is_empty() { + return Err(AppError::invalid("the file has no columns")); + } + Ok(Scanned { + schema, + chunk_rows, + data_bytes: file_size, + compression: None, + parquet_meta: None, + }) + } + } +} + +/// Inspect a columnar file: format, row/chunk counts, columns mapped to F31 +/// logical types, compression, nested fields and the editable-memory +/// estimate — everything the open dialog shows BEFORE any open. +pub fn inspect(path: &Path, ctx: Option<&JobCtx>) -> AppResult { + let format = detect_format(path)?; + let scanned = scan_source(path, format, ctx)?; + let options = ColumnarOpenOptions::default(); + let projection = build_projection(&scanned.schema, &options, false)?; + let n_rows: usize = scanned.chunk_rows.iter().sum(); + let file_size = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0); + let estimated_memory = estimate_memory( + n_rows as u64, + projection.cols.len() as u64, + scanned.data_bytes, + ); + + let columns = projection + .cols + .iter() + .map(|col| InspectedColumn { + name: col.name.clone(), + arrow_type: arrow_type_label(&col.data_type), + logical_type: col.logical, + nullable: col.nullable, + time_zone: col.time_zone.clone(), + nested: col.steps.len() > 1 || col.kind == OutKind::Json, + }) + .collect(); + let complex_fields = projection + .cols + .iter() + .filter(|c| c.kind == OutKind::Json) + .map(|c| c.name.clone()) + .collect(); + + Ok(ColumnarInspection { + format: format.wire_name().to_string(), + row_count: n_rows as u64, + chunk_count: scanned.chunk_rows.len() as u64, + compression: scanned.compression, + columns, + complex_fields, + estimated_memory, + needs_decision: estimated_memory > index::MEMORY_DECISION_THRESHOLD + || file_size > index::SIZE_DECISION_THRESHOLD, + file_size, + }) +} + +// --------------------------------------------------------------------------- +// Opening +// --------------------------------------------------------------------------- + +/// Everything [`crate::document::Document::from_columnar`] needs. +#[derive(Debug)] +pub struct ColumnarFile { + pub handle: ColumnarHandle, + pub headers: Vec, + /// Parallel to `headers`; column IDs are positional `c{i}` placeholders + /// that match the document's initial ID assignment. + pub schemas: Vec, +} + +/// Open a columnar file as an indexed READ-ONLY backing. Explode policies are +/// rejected here (they change the row count); use [`open_editable_rows`]. +/// Creates no on-disk caches — cancellation simply abandons the handle. +pub fn open_indexed( + path: &Path, + options: &ColumnarOpenOptions, + ctx: Option<&JobCtx>, +) -> AppResult { + let format = detect_format(path)?; + let scanned = scan_source(path, format, ctx)?; + let projection = build_projection(&scanned.schema, options, false)?; + let schemas = column_schemas(&projection.cols); + let headers: Vec = projection.cols.iter().map(|c| c.name.clone()).collect(); + + let mut chunk_starts = Vec::with_capacity(scanned.chunk_rows.len() + 1); + let mut total = 0usize; + chunk_starts.push(0); + for &rows in &scanned.chunk_rows { + total += rows; + chunk_starts.push(total); + } + let stats = scanned + .parquet_meta + .as_deref() + .and_then(|meta| extract_stats(meta, &projection.cols, &projection)); + if let Some(ctx) = ctx { + ctx.check()?; + } + + let editable_estimate = estimate_memory( + total as u64, + projection.cols.len() as u64, + scanned.data_bytes, + ); + let budget = if options.cache_budget_bytes == 0 { + DEFAULT_CACHE_BUDGET + } else { + options.cache_budget_bytes + }; + let handle = ColumnarHandle { + path: path.to_path_buf(), + format, + cols: projection.cols, + schemas: schemas.clone(), + chunk_starts, + n_rows: total, + source_check: util::stat_fingerprint(path), + editable_estimate, + stats, + cache: Mutex::new(BlockCache::new(budget)), + }; + Ok(ColumnarFile { + handle, + headers, + schemas, + }) +} + +// --------------------------------------------------------------------------- +// Editable materialisation (convert-to-editable + explode opens) +// --------------------------------------------------------------------------- + +/// A fully materialised editable table: text rows plus the schemas whose +/// null tokens carry the null-vs-empty distinction into the text plane. +#[derive(Debug)] +pub struct EditableTable { + pub headers: Vec, + /// Parallel to `headers`; positional `c{i}` column IDs. Columns that + /// contain at least one NULL get a collision-free null token. + pub schemas: Vec, + pub rows: Vec>, +} + +/// Deterministic per-column null token: the first of `NULL`, `NULL#1`, +/// `NULL#2`, … whose trimmed form never appears among the column's actual +/// (trimmed) values, so [`crate::schema::is_null_token`] can never +/// misclassify a real value. +fn pick_null_token(present: &dyn Fn(&str) -> bool) -> String { + if !present("NULL") { + return "NULL".to_string(); + } + let mut n = 1u64; + loop { + let candidate = format!("NULL#{n}"); + if !present(&candidate) { + return candidate; + } + n += 1; + } +} + +/// Turn `Option` rows into token-rendered text rows, assigning null tokens +/// to the schemas of columns that actually contain nulls. +fn tokenize_rows( + rows_opt: Vec>>, + schemas: &mut [ColumnSchema], +) -> Vec> { + let n_cols = schemas.len(); + let mut has_null = vec![false; n_cols]; + for row in &rows_opt { + for (c, cell) in row.iter().enumerate() { + if cell.is_none() { + has_null[c] = true; + } + } + } + let mut tokens: Vec> = vec![None; n_cols]; + for c in 0..n_cols { + if !has_null[c] { + continue; + } + let present = |candidate: &str| { + rows_opt.iter().any(|row| { + row.get(c) + .and_then(|cell| cell.as_deref()) + .is_some_and(|v| v.trim() == candidate) + }) + }; + let token = pick_null_token(&present); + schemas[c].null_tokens = vec![token.clone()]; + tokens[c] = Some(token); + } + rows_opt + .into_iter() + .map(|row| { + row.into_iter() + .enumerate() + .map(|(c, cell)| match cell { + Some(v) => v, + None => tokens[c].clone().unwrap_or_default(), + }) + .collect() + }) + .collect() +} + +/// Materialise an open indexed handle into editable rows + schemas (the +/// convert-to-editable payload). The caller is responsible for the explicit +/// memory check ([`ColumnarHandle::convert_needs_decision`]) and for applying +/// the result under a revision guard. +pub fn plan_editable(handle: &ColumnarHandle, ctx: Option<&JobCtx>) -> AppResult { + if let Some(ctx) = ctx { + ctx.set_total(handle.n_rows() as u64); + } + let mut rows_opt: Vec>> = Vec::with_capacity(handle.n_rows()); + let mut offset = 0u64; + loop { + let batch = handle.read_optional(offset, crate::tabular::DEFAULT_WINDOW, ctx)?; + if batch.is_empty() { + break; + } + let n = batch.len() as u64; + offset += n; + rows_opt.extend(batch); + if let Some(ctx) = ctx { + ctx.advance(n)?; + } + } + let mut schemas = handle.schemas().to_vec(); + let rows = tokenize_rows(rows_opt, &mut schemas); + Ok(EditableTable { + headers: handle.headers(), + schemas, + rows, + }) +} + +/// Open a columnar file straight to editable rows, honouring ALL policies +/// including exploding one list column into rows. The caller does the +/// explicit memory check first (via [`inspect`]). +pub fn open_editable_rows( + path: &Path, + options: &ColumnarOpenOptions, + ctx: Option<&JobCtx>, +) -> AppResult { + let format = detect_format(path)?; + let scanned = scan_source(path, format, ctx)?; + let projection = build_projection(&scanned.schema, options, true)?; + let mut schemas = column_schemas(&projection.cols); + let headers: Vec = projection.cols.iter().map(|c| c.name.clone()).collect(); + let n_rows: usize = scanned.chunk_rows.iter().sum(); + if let Some(ctx) = ctx { + ctx.set_total(n_rows as u64); + } + + let mut rows_opt: Vec>> = Vec::with_capacity(n_rows); + let mut push_batch = |batch: &RecordBatch| -> AppResult<()> { + render_batch_exploded(&projection.cols, batch, &mut rows_opt) + }; + match format { + ColumnarFormat::Parquet => { + let file = File::open(path)?; + let reader = ParquetRecordBatchReaderBuilder::try_new(file) + .map_err(arrow_err)? + .with_batch_size(BLOCK_ROWS) + .build() + .map_err(arrow_err)?; + for batch in reader { + if let Some(ctx) = ctx { + ctx.check()?; + } + let batch = batch.map_err(arrow_err)?; + let rows = batch.num_rows() as u64; + push_batch(&batch)?; + if let Some(ctx) = ctx { + ctx.advance(rows)?; + } + } + } + ColumnarFormat::ArrowFile => { + let reader = IpcFileReader::try_new(BufReader::new(File::open(path)?), None) + .map_err(arrow_err)?; + for batch in reader { + if let Some(ctx) = ctx { + ctx.check()?; + } + let batch = batch.map_err(arrow_err)?; + let rows = batch.num_rows() as u64; + push_batch(&batch)?; + if let Some(ctx) = ctx { + ctx.advance(rows)?; + } + } + } + ColumnarFormat::ArrowStream => { + let reader = IpcStreamReader::try_new(BufReader::new(File::open(path)?), None) + .map_err(arrow_err)?; + for batch in reader { + if let Some(ctx) = ctx { + ctx.check()?; + } + let batch = batch.map_err(arrow_err)?; + let rows = batch.num_rows() as u64; + push_batch(&batch)?; + if let Some(ctx) = ctx { + ctx.advance(rows)?; + } + } + } + } + + let rows = tokenize_rows(rows_opt, &mut schemas); + Ok(EditableTable { + headers, + schemas, + rows, + }) +} + +/// Like [`render_batch`], but honouring an Explode column: each source +/// record becomes one output row PER LIST ELEMENT (deterministic order); +/// null and empty lists both yield a single row with a null cell. +fn render_batch_exploded( + cols: &[OutCol], + batch: &RecordBatch, + out: &mut Vec, +) -> AppResult<()> { + let explode_at = cols.iter().position(|c| c.kind == OutKind::Explode); + let Some(explode_at) = explode_at else { + return render_batch(cols, batch, out); + }; + let cursors: Vec = cols + .iter() + .map(|c| LeafCursor::new(c, batch)) + .collect::>()?; + for row in 0..batch.num_rows() { + let mut base = Vec::with_capacity(cols.len()); + for (c, (col, cursor)) in cols.iter().zip(&cursors).enumerate() { + if c == explode_at { + base.push(None); // placeholder + } else { + base.push(cursor.cell(col, row)?); + } + } + let elements = explode_elements(&cursors[explode_at], row)?; + match elements { + None => out.push(base), + Some(elems) if elems.is_empty() => out.push(base), + Some(elems) => { + for element in elems { + let mut cloned = base.clone(); + cloned[explode_at] = element; + out.push(cloned); + } + } + } + } + Ok(()) +} + +/// The exploded cells for one record: `None` for a null list (or a null +/// ancestor), `Some(vec![...])` with one rendered cell per element. +fn explode_elements(cursor: &LeafCursor, row: usize) -> AppResult>>> { + for parent in &cursor.parents { + if parent.is_null(row) { + return Ok(None); + } + } + let leaf = cursor.leaf.as_ref(); + if leaf.is_null(row) { + return Ok(None); + } + let elems: ArrayRef = match leaf.data_type() { + DataType::List(_) => leaf + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))? + .value(row), + DataType::LargeList(_) => leaf + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))? + .value(row), + DataType::FixedSizeList(_, _) => leaf + .as_any() + .downcast_ref::() + .ok_or_else(|| arrow_err("array type mismatch"))? + .value(row), + _ => return Err(arrow_err("explode target is not a list")), + }; + let mut out = Vec::with_capacity(elems.len()); + for e in 0..elems.len() { + if is_primitive(elems.data_type()) { + out.push(render_primitive(elems.as_ref(), e)?); + } else if elems.is_null(e) { + out.push(None); + } else { + let value = json_at(elems.as_ref(), e)?; + out.push(Some(serde_json::to_string(&value).map_err(|err| { + arrow_err(format!("JSON encoding failed: {err}")) + })?)); + } + } + Ok(Some(out)) +} + +// --------------------------------------------------------------------------- +// TabularSource adapter +// --------------------------------------------------------------------------- + +/// [`TabularSource`] over an open [`ColumnarHandle`]: the `Option` plane — +/// columnar NULL is `None`, an empty string is `Some("")` — so export and +/// derived operations keep the distinction end to end. +pub struct ColumnarSource<'a> { + handle: &'a ColumnarHandle, +} + +impl<'a> ColumnarSource<'a> { + pub fn new(handle: &'a ColumnarHandle) -> ColumnarSource<'a> { + ColumnarSource { handle } + } +} + +impl TabularSource for ColumnarSource<'_> { + fn columns(&self) -> Vec { + self.handle.tabular_columns() + } + + fn row_count(&self) -> RowCountHint { + RowCountHint::Exact(self.handle.n_rows() as u64) + } + + fn read_rows( + &self, + offset: u64, + limit: usize, + ctx: Option<&JobCtx>, + ) -> AppResult> { + self.handle.read_optional(offset, limit, ctx) + } + + fn fingerprint(&self) -> ContentFingerprint { + match self.handle.source_check { + Some(fp) => ContentFingerprint::File(fp), + None => ContentFingerprint::Unknown, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Arc; + + use arrow::array::{ + ArrayRef, BooleanArray, Date32Array, Decimal128Array, DictionaryArray, Int64Array, + Int64Builder, ListArray, MapBuilder, StringArray, StringBuilder, TimestampMicrosecondArray, + UInt64Array, + }; + use arrow::buffer::NullBuffer; + use arrow::datatypes::{Field, Fields, Int64Type, Int8Type, Schema}; + use arrow::ipc::writer::{FileWriter, StreamWriter}; + use chrono::NaiveDate; + use parquet::arrow::ArrowWriter; + use parquet::basic::Compression; + use parquet::file::properties::WriterProperties; + + use crate::document::Document; + use crate::dto::{FilterCondition, FilterGroup, FilterNode, FilterOp}; + use crate::job::JobRegistry; + use crate::schema::{classify, CellState}; + use crate::tabular::DocumentSource; + + // ----- builders -------------------------------------------------------- + + fn batch_of(cols: Vec<(&str, ArrayRef)>) -> RecordBatch { + let fields: Vec = cols + .iter() + .map(|(n, a)| Field::new(*n, a.data_type().clone(), true)) + .collect(); + let schema = Arc::new(Schema::new(fields)); + RecordBatch::try_new(schema, cols.into_iter().map(|(_, a)| a).collect()).unwrap() + } + + fn write_parquet_with( + path: &Path, + batches: &[RecordBatch], + row_group_size: usize, + compression: Compression, + ) { + let props = WriterProperties::builder() + .set_max_row_group_row_count(Some(row_group_size)) + .set_compression(compression) + .build(); + let file = File::create(path).unwrap(); + let mut writer = ArrowWriter::try_new(file, batches[0].schema(), Some(props)).unwrap(); + for batch in batches { + writer.write(batch).unwrap(); + } + writer.close().unwrap(); + } + + fn write_parquet(path: &Path, batch: &RecordBatch) { + write_parquet_with( + path, + std::slice::from_ref(batch), + 1024 * 1024, + Compression::SNAPPY, + ); + } + + fn write_ipc_file(path: &Path, batches: &[RecordBatch]) { + let file = File::create(path).unwrap(); + let mut writer = FileWriter::try_new(file, batches[0].schema().as_ref()).unwrap(); + for batch in batches { + writer.write(batch).unwrap(); + } + writer.finish().unwrap(); + } + + fn write_ipc_stream(path: &Path, batches: &[RecordBatch]) { + let file = File::create(path).unwrap(); + let mut writer = StreamWriter::try_new(file, batches[0].schema().as_ref()).unwrap(); + for batch in batches { + writer.write(batch).unwrap(); + } + writer.finish().unwrap(); + } + + fn open(path: &Path) -> ColumnarFile { + open_indexed(path, &ColumnarOpenOptions::default(), None).unwrap() + } + + fn all_optional(handle: &ColumnarHandle) -> Vec { + handle.read_optional(0, handle.n_rows(), None).unwrap() + } + + fn all_text(handle: &ColumnarHandle) -> Vec> { + let mut out = Vec::new(); + handle + .visit(0..handle.n_rows(), &mut |_, row| { + out.push(row.to_vec()); + Ok(true) + }) + .unwrap(); + out + } + + fn cancelled_ctx(registry: &JobRegistry) -> JobCtx { + let ctx = registry.begin("test", None, |_| {}); + registry.cancel(ctx.id); + ctx + } + + fn days(y: i32, m: u32, d: u32) -> i32 { + let epoch = NaiveDate::from_ymd_opt(1970, 1, 1).unwrap(); + (NaiveDate::from_ymd_opt(y, m, d).unwrap() - epoch).num_days() as i32 + } + + fn cond(column: usize, op: FilterOp, value: &str) -> FilterNode { + FilterNode::Condition(FilterCondition { + column, + op, + value: value.to_string(), + case_sensitive: false, + }) + } + + fn and_group(nodes: Vec) -> FilterGroup { + FilterGroup { + conjunction: Conjunction::And, + nodes, + } + } + + // ----- type preservation matrix ---------------------------------------- + + #[test] + fn type_matrix_round_trips_extremes() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("t.parquet"); + let ts = NaiveDate::from_ymd_opt(2024, 1, 2) + .unwrap() + .and_hms_micro_opt(3, 4, 5, 123_456) + .unwrap(); + let batch = batch_of(vec![ + ("i", Arc::new(Int64Array::from(vec![i64::MAX, i64::MIN, 0]))), + ("u", Arc::new(UInt64Array::from(vec![u64::MAX, 0, 7]))), + ("f", Arc::new(Float64Array::from(vec![0.1, -1.5e300, 2.0]))), + ( + "dec", + Arc::new( + Decimal128Array::from(vec![150i128, -5, 0]) + .with_precision_and_scale(12, 2) + .unwrap(), + ), + ), + ("b", Arc::new(BooleanArray::from(vec![true, false, true]))), + ( + "d", + Arc::new(Date32Array::from(vec![ + days(2024, 1, 1), + 0, + days(1969, 12, 31), + ])), + ), + ( + "ts", + Arc::new(TimestampMicrosecondArray::from(vec![ + ts.and_utc().timestamp_micros(), + 0, + -1, + ])), + ), + ( + "s", + Arc::new(StringArray::from(vec!["x", "", "long text ünïcode"])), + ), + ( + "bin", + Arc::new(BinaryArray::from_opt_vec(vec![ + Some(b"\x0a\xff".as_ref()), + Some(b"".as_ref()), + None, + ])), + ), + ]); + write_parquet(&path, &batch); + + let file = open(&path); + assert_eq!( + file.headers, + ["i", "u", "f", "dec", "b", "d", "ts", "s", "bin"] + ); + let logicals: Vec = file.schemas.iter().map(|s| s.logical_type).collect(); + assert_eq!( + logicals, + [ + LogicalType::Integer, + LogicalType::Integer, + LogicalType::Float, + LogicalType::Decimal, + LogicalType::Boolean, + LogicalType::Date, + LogicalType::Datetime, + LogicalType::Text, + LogicalType::Text, + ] + ); + + let text = all_text(&file.handle); + assert_eq!(text[0][0], i64::MAX.to_string()); + assert_eq!(text[1][0], i64::MIN.to_string()); + assert_eq!(text[0][1], u64::MAX.to_string(), "u64::MAX lossless"); + assert_eq!(text[0][2], "0.1"); + assert_eq!(text[0][3], "1.50", "decimal keeps its scale"); + assert_eq!(text[1][3], "-0.05"); + assert_eq!(text[2][3], "0.00"); + assert_eq!(text[0][4], "true"); + assert_eq!(text[0][5], "2024-01-01"); + assert_eq!(text[2][5], "1969-12-31"); + assert_eq!(text[0][6], "2024-01-02T03:04:05.123456"); + assert_eq!(text[1][6], "1970-01-01T00:00:00"); + assert_eq!(text[2][6], "1969-12-31T23:59:59.999999"); + assert_eq!(text[2][7], "long text ünïcode"); + assert_eq!(text[0][8], "0aff", "binary renders as lowercase hex"); + + // Every rendered value classifies as VALID under its generated + // schema — the text plane is canonical, so it round-trips. + for row in &text { + for (c, cell) in row.iter().enumerate() { + if cell.is_empty() { + continue; + } + let state = classify(Some(cell), &file.schemas[c]); + assert!( + matches!(state, CellState::Valid(_)), + "column {c} cell {cell:?} classified {state:?}" + ); + } + } + + // The integer extremes survive typed classification exactly. + let CellState::Valid(TypedValue::Integer(v)) = + classify(Some(&text[0][1]), &file.schemas[1]) + else { + panic!("expected valid integer"); + }; + assert_eq!(v, i128::from(u64::MAX)); + } + + // ----- null vs empty ---------------------------------------------------- + + #[test] + fn null_vs_empty_string_round_trip_distinctly() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("n.parquet"); + let batch = batch_of(vec![ + ( + "s", + Arc::new(StringArray::from(vec![Some(""), None, Some("x")])), + ), + ( + "i", + Arc::new(Int64Array::from(vec![Some(5), None, Some(6)])), + ), + ]); + write_parquet(&path, &batch); + let file = open(&path); + + // Option plane: NULL is None, empty string is Some(""). + let rows = all_optional(&file.handle); + assert_eq!(rows[0][0], Some(String::new())); + assert_eq!(rows[1][0], None); + assert_eq!(rows[2][0], Some("x".to_string())); + assert_eq!(rows[1][1], None); + + // Text plane: both render as an empty cell (grid display). + let text = all_text(&file.handle); + assert_eq!(text[0][0], ""); + assert_eq!(text[1][0], ""); + + // The document-level tabular source keeps the distinction, so the + // export path sees the real null bit. + let doc = Document::from_columnar(1, Some(path.clone()), file); + let source = DocumentSource::new(&doc); + let rows = source.read_rows(0, 10, None).unwrap(); + assert_eq!(rows[0][0], Some(String::new())); + assert_eq!(rows[1][0], None); + assert!(source.has_header_row()); + } + + // ----- timestamps & time zones ------------------------------------------ + + #[test] + fn timestamp_time_zone_metadata_is_preserved() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("tz.parquet"); + let ts = NaiveDate::from_ymd_opt(2024, 6, 1) + .unwrap() + .and_hms_micro_opt(12, 30, 0, 250_000) + .unwrap(); + let micros = ts.and_utc().timestamp_micros(); + let batch = batch_of(vec![ + ( + "zoned", + Arc::new( + TimestampMicrosecondArray::from(vec![micros]).with_timezone("Europe/Berlin"), + ), + ), + ( + "naive", + Arc::new(TimestampMicrosecondArray::from(vec![micros])), + ), + ]); + write_parquet(&path, &batch); + let file = open(&path); + + assert_eq!( + file.schemas[0].time_zone.as_deref(), + Some("Europe/Berlin"), + "tz metadata maps to ColumnSchema.timeZone" + ); + assert_eq!(file.schemas[1].time_zone, None); + assert_eq!( + file.schemas[1].input_formats, + Some(vec!["%Y-%m-%dT%H:%M:%S%.f".to_string()]) + ); + + let text = all_text(&file.handle); + // Zoned: the UTC instant with an explicit Z (RFC 3339). + assert_eq!(text[0][0], "2024-06-01T12:30:00.250Z"); + // Naive: plain ISO wall time. + assert_eq!(text[0][1], "2024-06-01T12:30:00.250"); + + // Both classify back to the exact stored instant. + for (c, (cell, schema)) in text[0].iter().zip(&file.schemas).enumerate() { + let CellState::Valid(TypedValue::DateTime(parsed)) = classify(Some(cell), schema) + else { + panic!("column {c} did not classify as a datetime"); + }; + assert_eq!(parsed, ts, "column {c} round-trips the instant"); + } + } + + // ----- nested structs ---------------------------------------------------- + + #[test] + fn struct_flattening_is_deterministic_and_escapes_dots() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("s.parquet"); + let a: ArrayRef = Arc::new(Int64Array::from(vec![Some(1), Some(2), Some(3)])); + let bc: ArrayRef = Arc::new(StringArray::from(vec![Some("u"), Some("v"), Some("w")])); + let child_fields = Fields::from(vec![ + Field::new("a", DataType::Int64, true), + Field::new("b.c", DataType::Utf8, true), + ]); + // Row 1 has a NULL STRUCT: both flattened cells must be null even + // though the child arrays hold values there. + let validity = NullBuffer::from(vec![true, false, true]); + let s: ArrayRef = Arc::new(StructArray::new(child_fields, vec![a, bc], Some(validity))); + let batch = batch_of(vec![ + ("s", s), + ("d", Arc::new(Int64Array::from(vec![9, 8, 7]))), + ]); + write_parquet(&path, &batch); + + let first = open(&path); + let second = open(&path); + assert_eq!( + first.headers, + ["s.a", "s.b\\.c", "d"], + "stable escaped paths" + ); + assert_eq!(first.headers, second.headers, "deterministic across opens"); + assert_eq!( + first + .schemas + .iter() + .map(|s| s.logical_type) + .collect::>(), + second + .schemas + .iter() + .map(|s| s.logical_type) + .collect::>(), + ); + + let rows = all_optional(&first.handle); + assert_eq!(rows[0][0], Some("1".to_string())); + assert_eq!(rows[1][0], None, "null struct nulls its children"); + assert_eq!(rows[1][1], None); + assert_eq!(rows[1][2], Some("8".to_string())); + assert_eq!(rows[2][1], Some("w".to_string())); + } + + // ----- list/map policy matrix -------------------------------------------- + + fn list_map_batch() -> RecordBatch { + let list = ListArray::from_iter_primitive::(vec![ + Some(vec![Some(1), Some(2)]), + Some(vec![]), + None, + ]); + let mut map = MapBuilder::new(None, StringBuilder::new(), Int64Builder::new()); + map.keys().append_value("k"); + map.values().append_value(1); + map.append(true).unwrap(); + map.append(true).unwrap(); // empty map + map.append(false).unwrap(); // null map + let map = map.finish(); + batch_of(vec![ + ("id", Arc::new(Int64Array::from(vec![10, 20, 30]))), + ("l", Arc::new(list)), + ("m", Arc::new(map)), + ]) + } + + #[test] + fn list_and_map_policies_are_deterministic() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("p.parquet"); + write_parquet(&path, &list_map_batch()); + + // Default: preserve as JSON. Null and empty stay distinct. + let file = open(&path); + assert_eq!(file.headers, ["id", "l", "m"]); + assert_eq!(file.schemas[1].logical_type, LogicalType::Json); + let rows = all_optional(&file.handle); + assert_eq!(rows[0][1], Some("[1,2]".to_string())); + assert_eq!(rows[1][1], Some("[]".to_string())); + assert_eq!(rows[2][1], None, "null list is a null cell"); + assert_eq!(rows[0][2], Some("{\"k\":1}".to_string())); + assert_eq!(rows[1][2], Some("{}".to_string())); + assert_eq!(rows[2][2], None); + + // Reject drops the field from the schema deterministically. + let mut options = ColumnarOpenOptions::default(); + options + .field_policies + .insert("l".to_string(), ComplexPolicy::Reject); + let rejected = open_indexed(&path, &options, None).unwrap(); + assert_eq!(rejected.headers, ["id", "m"]); + + // Explode cannot re-number rows in an indexed backing. + let mut options = ColumnarOpenOptions::default(); + options + .field_policies + .insert("l".to_string(), ComplexPolicy::Explode); + let err = open_indexed(&path, &options, None).unwrap_err(); + assert!(err.to_string().contains("explode"), "{err}"); + + // Exploding a MAP is rejected outright. + let mut options = ColumnarOpenOptions::default(); + options + .field_policies + .insert("m".to_string(), ComplexPolicy::Explode); + let err = open_editable_rows(&path, &options, None).unwrap_err(); + assert!(err.to_string().contains("list"), "{err}"); + } + + #[test] + fn explode_multiplies_rows_in_the_editable_open() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("e.parquet"); + write_parquet(&path, &list_map_batch()); + + let mut options = ColumnarOpenOptions::default(); + options + .field_policies + .insert("l".to_string(), ComplexPolicy::Explode); + options + .field_policies + .insert("m".to_string(), ComplexPolicy::Reject); + let table = open_editable_rows(&path, &options, None).unwrap(); + assert_eq!(table.headers, ["id", "l"]); + // [1,2] explodes to two rows; the empty and null lists each yield one + // row whose exploded cell is null (rendered as the column's token). + let token = table.schemas[1].null_tokens.first().cloned().unwrap(); + assert_eq!( + table.rows, + vec![ + vec!["10".to_string(), "1".to_string()], + vec!["10".to_string(), "2".to_string()], + vec!["20".to_string(), token.clone()], + vec!["30".to_string(), token], + ] + ); + + // Two exploded lists are rejected (deterministic schemas only). + let list2 = ListArray::from_iter_primitive::(vec![ + Some(vec![Some(1)]), + Some(vec![Some(2)]), + Some(vec![Some(3)]), + ]); + let path2 = dir.path().join("e2.parquet"); + let batch = list_map_batch(); + let batch2 = batch_of(vec![ + ("id", batch.column(0).clone()), + ("l", batch.column(1).clone()), + ("l2", Arc::new(list2)), + ]); + write_parquet(&path2, &batch2); + let options = ColumnarOpenOptions { + complex_policy: ComplexPolicy::Explode, + ..Default::default() + }; + let err = open_editable_rows(&path2, &options, None).unwrap_err(); + assert!(err.to_string().contains("one list column"), "{err}"); + } + + // ----- convert-to-editable ------------------------------------------------ + + #[test] + fn convert_to_editable_assigns_collision_free_null_tokens() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("c.parquet"); + let batch = batch_of(vec![ + ( + "s", + Arc::new(StringArray::from(vec![ + Some("NULL"), + None, + Some("x"), + Some(""), + ])), + ), + ( + "i", + Arc::new(Int64Array::from(vec![Some(1), Some(2), None, Some(4)])), + ), + ("clean", Arc::new(Int64Array::from(vec![1, 2, 3, 4]))), + ]); + write_parquet(&path, &batch); + let file = open(&path); + let plan = plan_editable(&file.handle, None).unwrap(); + + // The literal string "NULL" exists in column s: the token escalates. + assert_eq!(plan.schemas[0].null_tokens, vec!["NULL#1".to_string()]); + assert_eq!(plan.rows[0][0], "NULL", "real value stays verbatim"); + assert_eq!(plan.rows[1][0], "NULL#1", "null renders as the token"); + assert_eq!(plan.rows[3][0], "", "empty string stays empty"); + // No collision in column i: the default token. + assert_eq!(plan.schemas[1].null_tokens, vec!["NULL".to_string()]); + assert_eq!(plan.rows[2][1], "NULL"); + // A column without nulls gets no token. + assert!(plan.schemas[2].null_tokens.is_empty()); + + // Applying the plan to the document keeps everything distinguishable. + let mut doc = Document::from_columnar(1, Some(path.clone()), file); + let ids: Vec = doc.column_ids().to_vec(); + doc.make_editable(plan.rows).unwrap(); + for (i, mut schema) in plan.schemas.into_iter().enumerate() { + schema.column_id = ids[i].clone(); + doc.set_column_schema(schema); + } + assert!(doc.is_editable()); + let s0 = doc.column_schema_at(0).unwrap(); + assert!(matches!(classify(Some("NULL#1"), s0), CellState::NullToken)); + assert!(matches!(classify(Some("NULL"), s0), CellState::Valid(_))); + assert!(matches!(classify(Some(""), s0), CellState::Empty)); + } + + #[test] + fn convert_memory_check_uses_the_columnar_estimate() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("m.parquet"); + let batch = batch_of(vec![("i", Arc::new(Int64Array::from_iter_values(0..100)))]); + write_parquet(&path, &batch); + let file = open(&path); + assert!(file.handle.editable_estimate() > 0); + assert!( + !file.handle.convert_needs_decision(), + "tiny file needs no gate" + ); + } + + // ----- arrow IPC file & stream --------------------------------------------- + + #[test] + fn arrow_ipc_file_and_stream_read_identically() { + let dir = tempfile::tempdir().unwrap(); + let b1 = batch_of(vec![ + ( + "i", + Arc::new(Int64Array::from(vec![Some(1), None, Some(3)])), + ), + ( + "s", + Arc::new(StringArray::from(vec![Some("a"), Some(""), None])), + ), + ]); + let b2 = batch_of(vec![ + ( + "i", + Arc::new(Int64Array::from(vec![Some(4), Some(5), Some(6)])), + ), + ( + "s", + Arc::new(StringArray::from(vec![Some("d"), Some("e"), Some("f")])), + ), + ]); + let file_path = dir.path().join("t.arrow"); + let stream_path = dir.path().join("t.arrows"); + write_ipc_file(&file_path, &[b1.clone(), b2.clone()]); + write_ipc_stream(&stream_path, &[b1, b2]); + + assert_eq!( + detect_format(&file_path).unwrap(), + ColumnarFormat::ArrowFile + ); + assert_eq!( + detect_format(&stream_path).unwrap(), + ColumnarFormat::ArrowStream + ); + + let f = open(&file_path); + let s = open(&stream_path); + assert_eq!(f.handle.n_rows(), 6); + assert_eq!(all_optional(&f.handle), all_optional(&s.handle)); + + // A window crossing the batch boundary reads contiguously. + let window = f.handle.read_optional(2, 3, None).unwrap(); + assert_eq!(window.len(), 3); + assert_eq!(window[0][0], Some("3".to_string())); + assert_eq!(window[1][0], Some("4".to_string())); + // Offset past the end yields an empty batch (tabular contract). + assert!(f.handle.read_optional(100, 5, None).unwrap().is_empty()); + // Null vs empty survives both containers. + let rows = all_optional(&s.handle); + assert_eq!(rows[1][0], None); + assert_eq!(rows[1][1], Some(String::new())); + assert_eq!(rows[2][1], None); + } + + #[test] + fn dictionary_columns_render_their_values() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("d.arrow"); + let dict: DictionaryArray = vec![Some("a"), Some("b"), Some("a"), None] + .into_iter() + .collect(); + let batch = batch_of(vec![("k", Arc::new(dict))]); + write_ipc_file(&path, &[batch]); + let file = open(&path); + assert_eq!(file.schemas[0].logical_type, LogicalType::Text); + let rows = all_optional(&file.handle); + assert_eq!(rows[0][0], Some("a".to_string())); + assert_eq!(rows[1][0], Some("b".to_string())); + assert_eq!(rows[3][0], None); + } + + // ----- statistics pruning ----------------------------------------------- + + /// 20 ordered rows in row groups of 4: groups [0..4), [4..8), … [16..20). + fn stats_doc(dir: &tempfile::TempDir) -> Document { + let path = dir.path().join("stats.parquet"); + let batch = batch_of(vec![ + ("n", Arc::new(Int64Array::from_iter_values(0..20))), + ( + "d", + Arc::new(Date32Array::from_iter_values( + (0..20).map(|i| days(2024, 1, 1) + i), + )), + ), + ]); + write_parquet_with( + &path, + std::slice::from_ref(&batch), + 4, + Compression::UNCOMPRESSED, + ); + let file = open(&path); + Document::from_columnar(1, Some(path), file) + } + + #[test] + // The expected values genuinely ARE one-element range lists here. + #[allow(clippy::single_range_in_vec_init)] + fn row_group_stats_skip_groups_and_match_the_full_scan() { + let dir = tempfile::tempdir().unwrap(); + let doc = stats_doc(&dir); + + // Range on the int column: only the last group can match n > 15. + let spec = and_group(vec![cond(0, FilterOp::Gt, "15")]); + let ranges = doc.filter_scan_ranges(&spec).expect("pruning applies"); + assert_eq!(ranges, [16..20], "first four groups proven impossible"); + + let matched = crate::filter::matching_rows(&doc, &spec).unwrap(); + assert_eq!(matched, vec![16, 17, 18, 19]); + + // Equality: only the group containing 6. + let spec = and_group(vec![cond(0, FilterOp::Equals, "6")]); + assert_eq!(doc.filter_scan_ranges(&spec).as_deref(), Some(&[4..8][..])); + assert_eq!(crate::filter::matching_rows(&doc, &spec).unwrap(), vec![6]); + + // Date range prunes chronologically. + let spec = and_group(vec![cond(1, FilterOp::Lt, "2024-01-03")]); + assert_eq!(doc.filter_scan_ranges(&spec).as_deref(), Some(&[0..4][..])); + assert_eq!( + crate::filter::matching_rows(&doc, &spec).unwrap(), + vec![0, 1] + ); + + // ACCEPTANCE: the filtered read returns exactly the same values as + // an unfiltered full scan for the matching rows. + let spec = and_group(vec![cond(0, FilterOp::Gte, "13")]); + let matched = crate::filter::matching_rows(&doc, &spec).unwrap(); + let mut brute = Vec::new(); + let mut brute_values = Vec::new(); + doc.visit_rows(0..doc.n_rows(), &mut |i, row| { + if row[0].parse::().is_ok_and(|v| v >= 13) { + brute.push(i); + brute_values.push(row.to_vec()); + } + Ok(true) + }) + .unwrap(); + assert_eq!(matched, brute, "pruned scan finds identical rows"); + assert_eq!( + doc.fetch_rows(&matched).unwrap(), + brute_values, + "identical values for matching rows" + ); + } + + #[test] + fn stats_pruning_falls_back_gracefully() { + let dir = tempfile::tempdir().unwrap(); + let mut doc = stats_doc(&dir); + + // OR specs contribute no required conditions. + let spec = FilterGroup { + conjunction: Conjunction::Or, + nodes: vec![cond(0, FilterOp::Gt, "15"), cond(0, FilterOp::Lt, "2")], + }; + assert_eq!(doc.filter_scan_ranges(&spec), None); + assert_eq!( + crate::filter::matching_rows(&doc, &spec).unwrap(), + vec![0, 1, 16, 17, 18, 19] + ); + + // Text-op conditions are ineligible. + let spec = and_group(vec![cond(0, FilterOp::Contains, "1")]); + assert_eq!(doc.filter_scan_ranges(&spec), None); + + // A schema whose parsing moved since the open (locale change) makes + // the stats domain untrustworthy: full scan, still correct. + let mut changed = doc.column_schema_at(0).unwrap().clone(); + changed.locale = Some("de-DE".to_string()); + doc.set_column_schema(changed); + let spec = and_group(vec![cond(0, FilterOp::Gt, "15")]); + assert_eq!(doc.filter_scan_ranges(&spec), None); + assert_eq!( + crate::filter::matching_rows(&doc, &spec).unwrap(), + vec![16, 17, 18, 19] + ); + } + + // ----- document integration ------------------------------------------------ + + #[test] + fn columnar_documents_behave_like_indexed_read_only_documents() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("doc.parquet"); + let batch = batch_of(vec![ + ("a", Arc::new(Int64Array::from(vec![1, 2, 3, 4]))), + ("b", Arc::new(StringArray::from(vec!["w", "x", "y", "z"]))), + ]); + write_parquet(&path, &batch); + let file = open(&path); + let mut doc = Document::from_columnar(7, Some(path), file); + + assert_eq!(doc.backing_name(), "indexedReadOnly"); + assert!(!doc.is_editable()); + assert!(matches!(doc.ensure_editable(), Err(AppError::ReadOnly))); + assert_eq!(doc.n_rows(), 4); + assert_eq!(doc.n_cols(), 2); + assert_eq!(doc.headers(), ["a", "b"]); + assert!( + doc.has_header_row(), + "real flattened names, never synthetic" + ); + assert_eq!( + doc.column_schema_at(0).map(|s| s.logical_type), + Some(LogicalType::Integer) + ); + + // Random-access fetch in caller order (view sorts use this). + let rows = doc.fetch_rows(&[2, 0, 2]).unwrap(); + assert_eq!(rows[0][1], "y"); + assert_eq!(rows[1][1], "w"); + assert_eq!(rows[2][1], "y"); + assert!(doc.fetch_rows(&[99]).is_err(), "out of range errors"); + + // The filter view composes over the columnar backing. + doc.set_filter(vec![1, 3]).unwrap(); + assert_eq!(doc.visible_len(), 2); + let resp = doc.get_rows(0, 10).unwrap(); + assert_eq!(resp.rows.len(), 2); + assert_eq!(resp.rows[0][1], "x"); + assert_eq!(resp.rows[1][1], "z"); + } + + // ----- cache bounds ----------------------------------------------------- + + #[test] + fn block_cache_stays_within_its_budget() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("big.parquet"); + let n = (BLOCK_ROWS * 2 + 10) as i64; + let batch = batch_of(vec![("i", Arc::new(Int64Array::from_iter_values(0..n)))]); + write_parquet(&path, &batch); + + // A one-byte budget: every newly decoded block evicts the previous. + let options = ColumnarOpenOptions { + cache_budget_bytes: 1, + ..Default::default() + }; + let file = open_indexed(&path, &options, None).unwrap(); + let handle = &file.handle; + for block in 0..3 { + handle + .read_optional((block * BLOCK_ROWS) as u64, 1, None) + .unwrap(); + assert_eq!(handle.cached_blocks(), 1, "budget keeps exactly one block"); + } + + // The default budget comfortably holds all three blocks. + let file = open(&path); + let handle = &file.handle; + let all = all_optional(handle); + assert_eq!(all.len(), n as usize); + assert_eq!(all[BLOCK_ROWS * 2 + 9][0], Some((n - 1).to_string())); + assert_eq!(handle.cached_blocks(), 3); + assert!(handle.cached_bytes() <= DEFAULT_CACHE_BUDGET); + } + + // ----- cancellation ----------------------------------------------------- + + #[test] + fn cancel_stops_open_inspect_and_convert_without_leftovers() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("c.parquet"); + let batch = batch_of(vec![("i", Arc::new(Int64Array::from_iter_values(0..64)))]); + write_parquet(&path, &batch); + let registry = JobRegistry::default(); + + let before: Vec<_> = std::fs::read_dir(dir.path()) + .unwrap() + .map(|e| e.unwrap().file_name()) + .collect(); + + let ctx = cancelled_ctx(®istry); + assert!(matches!( + open_indexed(&path, &ColumnarOpenOptions::default(), Some(&ctx)), + Err(AppError::Cancelled) + )); + let ctx = cancelled_ctx(®istry); + assert!(matches!( + inspect(&path, Some(&ctx)), + Err(AppError::Cancelled) + )); + let ctx = cancelled_ctx(®istry); + assert!(matches!( + open_editable_rows(&path, &ColumnarOpenOptions::default(), Some(&ctx)), + Err(AppError::Cancelled) + )); + + let file = open(&path); + let ctx = cancelled_ctx(®istry); + assert!(matches!( + plan_editable(&file.handle, Some(&ctx)), + Err(AppError::Cancelled) + )); + + // The read side never writes: no caches or partial outputs appear. + let after: Vec<_> = std::fs::read_dir(dir.path()) + .unwrap() + .map(|e| e.unwrap().file_name()) + .collect(); + assert_eq!(before, after, "cancellation leaves the directory untouched"); + } + + // ----- inspection --------------------------------------------------------- + + #[test] + fn inspect_reports_shape_types_and_compression() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("i.parquet"); + let batch = batch_of(vec![ + ("n", Arc::new(Int64Array::from_iter_values(0..10))), + ( + "l", + Arc::new(ListArray::from_iter_primitive::(vec![ + Some( + vec![Some(1)] + ); + 10 + ])), + ), + ]); + write_parquet_with(&path, std::slice::from_ref(&batch), 4, Compression::SNAPPY); + + let report = inspect(&path, None).unwrap(); + assert_eq!(report.format, "parquet"); + assert_eq!(report.row_count, 10); + assert_eq!(report.chunk_count, 3, "10 rows in groups of 4"); + assert!( + report + .compression + .as_deref() + .unwrap_or("") + .contains("SNAPPY"), + "codec surfaces: {:?}", + report.compression + ); + assert_eq!(report.columns.len(), 2); + assert_eq!(report.columns[0].logical_type, LogicalType::Integer); + assert_eq!(report.columns[1].logical_type, LogicalType::Json); + assert!(report.columns[1].nested); + assert_eq!(report.complex_fields, vec!["l".to_string()]); + assert!(report.estimated_memory > 0); + assert!(!report.needs_decision); + + // Arrow IPC file: same surface, no compression, batch chunks. + let arrow_path = dir.path().join("i.arrow"); + write_ipc_file(&arrow_path, &[batch.clone(), batch]); + let report = inspect(&arrow_path, None).unwrap(); + assert_eq!(report.format, "arrowFile"); + assert_eq!(report.row_count, 20); + assert_eq!(report.chunk_count, 2); + assert_eq!(report.compression, None); + } +} diff --git a/src-tauri/src/tabular.rs b/src-tauri/src/tabular.rs index b97171a..5be5e41 100644 --- a/src-tauri/src/tabular.rs +++ b/src-tauri/src/tabular.rs @@ -11,11 +11,15 @@ //! //! Cells are `Option`: `None` means the field is MISSING from the //! record (a short JSONL object, an absent Excel cell, SQL `NULL`), while -//! `Some("")` is a present-but-empty value. Document-backed sources always -//! produce `Some` — the grid is rectangular by construction (import-time -//! padding is recorded in [`crate::parse::ImportInfo`], not in the cells) — -//! but the distinction is part of the contract so future sources can carry -//! it and sinks can decide how to narrow it (CSV writes missing as empty). +//! `Some("")` is a present-but-empty value. Text-document-backed sources +//! (editable / F10 indexed) always produce `Some` — the grid is rectangular +//! by construction (import-time padding is recorded in +//! [`crate::parse::ImportInfo`], not in the cells). A COLUMNAR-backed +//! document (F32 Parquet/Arrow) carries the distinction for real: a columnar +//! NULL reads as `None`, an empty string as `Some("")` — its rectangular +//! text plane renders NULL as an empty cell, but this contract keeps the +//! null bit so export and derivation round-trip it. Sinks decide how to +//! narrow it (CSV writes missing as empty). //! //! ## Contract scope and access model //! @@ -261,6 +265,12 @@ impl TabularSource for DocumentSource<'_> { if limit == 0 { return Ok(Vec::new()); } + // Columnar documents (F32) read through the handle's Option plane so + // a columnar NULL stays `None` (distinct from `Some("")`); the text + // plane below would flatten both to an empty cell. + if let Some(handle) = self.doc.columnar_handle() { + return handle.read_optional(offset, limit, ctx); + } let n = self.doc.n_rows(); let start = usize::try_from(offset).unwrap_or(usize::MAX).min(n); let end = start.saturating_add(limit).min(n); diff --git a/src/App.tsx b/src/App.tsx index 1fa6e34..1804ef3 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -40,6 +40,8 @@ import { Close } from "./components/Icons"; import { JoinDialog } from "./components/JoinDialog"; import { JsonExportDialog } from "./components/JsonExportDialog"; import { JsonImportDialog } from "./components/JsonImportDialog"; +import { ColumnarExportDialog } from "./components/ColumnarExportDialog"; +import { ParquetInspectDialog } from "./components/ParquetInspectDialog"; import { ProfilesDialog } from "./components/ProfilesDialog"; import { ProfileSuggestionBar } from "./components/ProfileSuggestionBar"; import { OpenModeDialog } from "./components/OpenModeDialog"; @@ -86,6 +88,7 @@ export default function App() { const activeModal = useStore((s) => s.activeModal); const setModal = useStore((s) => s.setModal); const jsonImportPath = useStore((s) => s.jsonImport?.path ?? null); + const columnarOpenPath = useStore((s) => s.columnarOpen?.path ?? null); const diagnosticsOpen = useStore((s) => s.diagnosticsOpen); const changesOpen = useStore((s) => s.changesOpen); const annotationsPanelOpen = useStore((s) => s.annotationsPanelOpen); @@ -352,7 +355,9 @@ export default function App() { {activeModal === "highlight" && setModal(null)} />} {activeModal === "pasteSpecial" && setModal(null)} />} {activeModal === "jsonExport" && setModal(null)} />} + {activeModal === "columnarExport" && setModal(null)} />} {jsonImportPath && } + {columnarOpenPath && } diff --git a/src/components/ColumnarExportDialog.tsx b/src/components/ColumnarExportDialog.tsx new file mode 100644 index 0000000..4950d9c --- /dev/null +++ b/src/components/ColumnarExportDialog.tsx @@ -0,0 +1,317 @@ +import { useEffect, useMemo, useState } from "react"; + +import { + columnarFormatLabel, + compressionLabel, + defaultColumnarExportOptions, +} from "../lib/columnar"; +import { scopeChoices, scopeKey } from "../lib/export"; +import { formatBytes } from "../lib/save"; +import * as api from "../lib/tauri"; +import { useActiveMeta, useStore } from "../store/useStore"; +import type { + ColumnarCompression, + ColumnarExportOptions, + ColumnarFormat, + ExportScope, + ScopeCounts, +} from "../types"; +import { Modal } from "./Modal"; + +const FORMATS: ColumnarFormat[] = ["parquet", "arrowFile", "arrowStream"]; +const COMPRESSIONS: ColumnarCompression[] = ["uncompressed", "snappy", "zstd"]; + +/** Bound the per-column warning list rendered on the done phase (DOM only). */ +const MAX_WARNING_ROWS = 200; + +/** + * Parquet / Arrow export (F32). Format (Parquet / Arrow IPC file = Feather v2 / + * Arrow IPC stream), Parquet compression + row-group size, typed vs verbatim + * emission, and any CEESVEE export scope. Typed export maps declared F31 types + * to arrow types; cells that can't be represented export as NULL and are + * counted — the done phase surfaces those per-column invalid-cell counts. + * Exports never touch the document's save point. + */ +export function ColumnarExportDialog({ onClose }: { onClose: () => void }) { + const meta = useActiveMeta(); + const runExport = useStore((s) => s.runColumnarExport); + const clearExport = useStore((s) => s.clearColumnarExport); + const result = useStore((s) => s.columnarExportResult); + const filtered = useStore((s) => s.tabs.find((t) => t.id === s.activeId)?.filtered ?? false); + const selectedRows = useStore((s) => s.selectedRows); + const selectedCols = useStore((s) => s.selectedCols); + const selectionRect = useStore((s) => s.selectionPhysicalRect)(); + const viewSorted = meta?.viewSorted ?? false; + + const [opts, setOpts] = useState(() => defaultColumnarExportOptions()); + + const choices = useMemo( + () => scopeChoices(filtered, selectionRect, selectedRows, selectedCols, viewSorted), + [filtered, selectionRect, selectedRows, selectedCols, viewSorted], + ); + const [scopeIdx, setScopeIdx] = useState(0); + const scope: ExportScope = (choices[scopeIdx] ?? choices[0]).scope; + + const [counts, setCounts] = useState(null); + + // Clear any stale result from a previous export when the dialog opens. + useEffect(() => { + clearExport(); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []); + + // Show the expected shape for the chosen scope before anything is written. + useEffect(() => { + if (!meta) return; + let stale = false; + setCounts(null); + api + .exportScopeCounts(meta.id, scope) + .then((c) => { + if (!stale) setCounts(c); + }) + .catch(() => undefined); + return () => { + stale = true; + }; + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [meta?.id, meta?.revision, scopeKey(scope), scopeIdx]); + + if (!meta) return null; + const patch = (p: Partial) => setOpts((o) => ({ ...o, ...p })); + + const isParquet = opts.format === "parquet"; + const report = result.report; + const running = result.running; + + const closeAndClear = () => { + clearExport(); + onClose(); + }; + + // ----- done phase: report + invalid-cell warning surface ------------------- + if (report) { + const shownWarnings = report.columnWarnings.slice(0, MAX_WARNING_ROWS); + const hiddenWarnings = report.columnWarnings.length - shownWarnings.length; + return ( + + Done + + } + > +
+
+
Format
+
{columnarFormatLabel(report.format)}
+
Rows written
+
{report.rows.toLocaleString()}
+
Columns
+
{report.columns.toLocaleString()}
+
File size
+
{formatBytes(report.bytes)}
+
+ + {report.invalidCells > 0 ? ( +
+

+ {report.invalidCells.toLocaleString()} cell + {report.invalidCells === 1 ? "" : "s"} could not be represented under the declared + types and were written as NULL: +

+
+ + + + + + + + + {shownWarnings.map((w) => ( + + + + + ))} + +
ColumnInvalid cells → NULL
+ {w.name} + + {w.invalidCells.toLocaleString()} +
+
+ {hiddenWarnings > 0 && ( +

+ + {hiddenWarnings.toLocaleString()} more column + {hiddenWarnings === 1 ? "" : "s"} with warnings. +

+ )} +

+ Turn off typed export to write these columns as text verbatim instead. +

+
+ ) : ( +

+ All cells exported cleanly — no invalid values. +

+ )} +
+ + ); + } + + // ----- config phase -------------------------------------------------------- + const doExport = () => { + if (running) return; + void runExport(opts, scope); + }; + + return ( + + + + + } + > +
+ + + + + {isParquet ? ( + <> + + + + + patch({ rowGroupRows: Math.max(0, Number(e.target.value) || 0) })} + className={inputCls} + /> + +

+ 0 uses the writer default. Smaller row groups let a later read skip more of the file + via row-group statistics. +

+ + ) : ( +

+ Arrow IPC output is always uncompressed. + {opts.format === "arrowFile" && + " The Arrow IPC file format is also known as Feather v2."} +

+ )} + + + + + +

+ {counts + ? `Will write ${counts.rows.toLocaleString()} data row${counts.rows === 1 ? "" : "s"} × ${counts.cols} column${counts.cols === 1 ? "" : "s"}` + : "Counting…"} +

+ +
+ +
+ + + +
+ +

+ Typed export preserves nulls (distinct from empty strings), integer widths (i64/u64), + decimal precision/scale, and timestamp timezones. Cells that don't fit the declared type + export as NULL and are reported after the write. +

+ + {result.error &&

{result.error}

} +
+ + ); +} + +function Row({ label, children }: { label: string; children: React.ReactNode }) { + return ( +
+ {label} + {children} +
+ ); +} + +const btnGhost = + "rounded px-3 py-1.5 text-sm text-zinc-600 hover:bg-zinc-100 dark:text-zinc-300 dark:hover:bg-zinc-800"; +const btnPrimary = "rounded bg-violet-600 px-3 py-1.5 text-sm text-white hover:bg-violet-500"; +const selectCls = + "rounded border border-zinc-300 bg-transparent px-2 py-1 text-sm outline-none focus:border-violet-500 dark:border-zinc-700"; +const inputCls = + "w-32 rounded border border-zinc-300 bg-transparent px-2 py-1 text-right text-sm tabular-nums outline-none focus:border-violet-500 dark:border-zinc-700"; diff --git a/src/components/OpenModeDialog.tsx b/src/components/OpenModeDialog.tsx index dda03e0..9459281 100644 --- a/src/components/OpenModeDialog.tsx +++ b/src/components/OpenModeDialog.tsx @@ -87,8 +87,13 @@ export function OpenModeDialog() { />

- {formatBytes(indexing.processed)} - {indexing.total ? ` of ${formatBytes(indexing.total)}` : ""} scanned + {indexing.unit === "rows" + ? `${indexing.processed.toLocaleString()}${ + indexing.total ? ` of ${indexing.total.toLocaleString()}` : "" + } rows` + : `${formatBytes(indexing.processed)}${ + indexing.total ? ` of ${formatBytes(indexing.total)}` : "" + } scanned`} {pct !== null ? ` · ${pct}%` : ""}

diff --git a/src/components/ParquetInspectDialog.tsx b/src/components/ParquetInspectDialog.tsx new file mode 100644 index 0000000..cd4c91c --- /dev/null +++ b/src/components/ParquetInspectDialog.tsx @@ -0,0 +1,342 @@ +import { useMemo, useState } from "react"; + +import { + chunkUnitLabel, + columnDepth, + columnarFormatLabel, + columnarOpenPlan, + defaultColumnarOpenOptions, + effectivePolicy, + estimatedMemoryLabel, + leafName, + setFieldPolicy, +} from "../lib/columnar"; +import { formatBytes } from "../lib/save"; +import { LOGICAL_TYPE_LABELS } from "../lib/schema"; +import { useStore } from "../store/useStore"; +import type { ColumnarOpenOptions, ComplexPolicy } from "../types"; +import { Modal } from "./Modal"; + +/** + * Upper bound on how many schema rows the dialog renders at once. The open + * itself is unaffected (the backend maps every column); this only keeps the + * DOM bounded, per the "bounded windows to React only" invariant — a wide + * columnar file can be thousands of flattened columns. + */ +const MAX_SCHEMA_ROWS = 300; + +/** + * Parquet / Arrow inspect + open dialog (F32). Self-driven by the + * `columnarOpen` store slice, so opening a `.parquet` / `.arrow` / `.feather` + * / `.ipc` file shows it automatically. Reports the container, row and + * row-group/batch counts, compression, the F31-mapped schema (nested fields + * indented), the editable-memory estimate, and per-field policy pickers for + * complex (list/map/struct-as-JSON) fields. The two open modes — indexed + * read-only vs converted-editable — mirror the read side's constraints + * (exploding a list forces an editable open, at most one per open). + */ +export function ParquetInspectDialog() { + const st = useStore((s) => s.columnarOpen); + const dismiss = useStore((s) => s.dismissColumnarInspect); + const openIndexed = useStore((s) => s.columnarOpenIndexed); + const openEditable = useStore((s) => s.columnarOpenEditable); + + const [options, setOptions] = useState(() => defaultColumnarOpenOptions()); + // Two-step acknowledgement for a large editable open (mirrors OpenModeDialog). + const [ackEditable, setAckEditable] = useState(false); + + const inspection = st?.inspection ?? null; + const plan = useMemo( + () => (inspection ? columnarOpenPlan(inspection, options) : null), + [inspection, options], + ); + + if (!st) return null; + + const shownColumns = inspection ? inspection.columns.slice(0, MAX_SCHEMA_ROWS) : []; + const hiddenColumns = (inspection?.columns.length ?? 0) - shownColumns.length; + const needsDecision = inspection?.needsDecision ?? false; + const blocked = (plan?.errors.length ?? 0) > 0; + const indexedDisabled = blocked || plan?.requiresEditable === true || !inspection; + const isArrow = inspection?.format !== "parquet"; + + const setPolicy = (path: string, policy: ComplexPolicy) => { + setOptions((o) => setFieldPolicy(o, path, policy)); + setAckEditable(false); + }; + const setDefaultPolicy = (policy: ComplexPolicy) => { + setOptions((o) => ({ ...o, complexPolicy: policy })); + setAckEditable(false); + }; + + const doEditable = () => { + if (blocked) return; + if (needsDecision && !ackEditable) { + setAckEditable(true); + return; + } + void openEditable(options, needsDecision); + }; + + return ( + + + + + + } + > +
+ {st.loading && !inspection && ( +

Inspecting file…

+ )} + {st.error &&

{st.error}

} + + {inspection && ( + <> + {/* Summary */} +
+ + Format:{" "} + {columnarFormatLabel(inspection.format)} + + + {inspection.rowCount.toLocaleString()} row + {inspection.rowCount === 1 ? "" : "s"} + + · + + {inspection.chunkCount.toLocaleString()}{" "} + {chunkUnitLabel(inspection.format, inspection.chunkCount)} + + {inspection.compression && ( + <> + · + codec {inspection.compression} + + )} + · + {formatBytes(inspection.fileSize)} on disk + · + + {inspection.columns.length.toLocaleString()} column + {inspection.columns.length === 1 ? "" : "s"} + +
+ + {isArrow && ( +

+ Arrow IPC file is also known as Feather v2 — + the same container. +

+ )} + + {/* Schema table with indented nesting */} +
+
+ + + + + + + + + + + {shownColumns.map((c) => { + const depth = columnDepth(c.name); + return ( + + + + + + + ); + })} + +
ColumnLogical typeArrow typeNull
+ {depth > 0 && ↳ } + {depth > 0 ? leafName(c.name) : c.name} + {c.nested && ( + + nested + + )} + + {LOGICAL_TYPE_LABELS[c.logicalType]} + {c.timeZone && ( + + ({c.timeZone}) + + )} + + {c.arrowType} + {c.nullable ? "yes" : "—"}
+
+ {hiddenColumns > 0 && ( +

+ + {hiddenColumns.toLocaleString()} more column + {hiddenColumns === 1 ? "" : "s"} not shown (all open). +

+ )} +
+ + {/* Complex-field policies */} + {inspection.complexFields.length > 0 && ( +
+
+ Default policy + setDefaultPolicy(v as ComplexPolicy)} + /> +
+
+ {inspection.complexFields.map((path) => ( +
+ + {path} + + +
+ ))} +
+

+ Exploding a list multiplies the record into one row per element — editable open + only, and one field at a time. +

+
+ )} + + {/* Open-plan errors / editable-only notice */} + {plan?.errors.map((e, i) => ( +

+ {e} +

+ ))} + {plan?.requiresEditable && plan.errors.length === 0 && ( +

+ {plan.indexedDisabledReason} +

+ )} + + {/* Memory estimate */} +
+
+ + Estimated memory if fully editable + + ~{estimatedMemoryLabel(inspection)} +
+ {needsDecision ? ( +

+ This is large — read-only (indexed) keeps memory bounded and still supports + browsing, find, filter, export and profiling. Converting to editable loads it all + into memory. +

+ ) : ( +

+ Read-only (indexed) streams rows on demand; convert to editable to change cells. + Either way the source file is never written over — an edited copy saves to a new + destination. +

+ )} + {needsDecision && ackEditable && ( +

+ Opening editable may exhaust memory. Click again to proceed anyway. +

+ )} +
+ + )} +
+ + ); +} + +function Section({ label, children }: { label: string; children: React.ReactNode }) { + return ( +
+

{label}

+ {children} +
+ ); +} + +function Segmented({ + value, + options, + onChange, +}: { + value: string; + options: { value: string; label: string }[]; + onChange: (value: string) => void; +}) { + return ( +
+ {options.map((o) => ( + + ))} +
+ ); +} + +const btnGhost = + "rounded px-3 py-1.5 text-sm text-zinc-600 hover:bg-zinc-100 dark:text-zinc-300 dark:hover:bg-zinc-800"; +const selectCls = + "rounded border border-zinc-300 bg-transparent px-2 py-1 text-xs outline-none focus:border-violet-500 dark:border-zinc-700"; diff --git a/src/lib/columnar.test.ts b/src/lib/columnar.test.ts new file mode 100644 index 0000000..a9ef3c1 --- /dev/null +++ b/src/lib/columnar.test.ts @@ -0,0 +1,185 @@ +import { describe, expect, it } from "vitest"; + +import { + chunkUnitLabel, + columnDepth, + columnarFormatExtension, + columnarFormatLabel, + columnarOpenPlan, + compressionLabel, + defaultColumnarExportOptions, + defaultColumnarOpenOptions, + effectivePolicy, + estimatedMemoryLabel, + explodeFields, + isColumnarPath, + leafName, + setFieldPolicy, + suggestColumnarFileName, +} from "./columnar"; +import type { ColumnarInspection, ColumnarOpenOptions } from "../types"; + +function inspection(overrides: Partial = {}): ColumnarInspection { + return { + format: "parquet", + rowCount: 100, + chunkCount: 2, + compression: "SNAPPY", + columns: [], + complexFields: [], + estimatedMemory: 5 * 1024 * 1024, + needsDecision: false, + fileSize: 1024, + ...overrides, + }; +} + +describe("format & compression labels", () => { + it("labels each container, noting Feather v2 for the Arrow IPC file", () => { + expect(columnarFormatLabel("parquet")).toBe("Apache Parquet"); + expect(columnarFormatLabel("arrowFile")).toContain("Feather v2"); + expect(columnarFormatLabel("arrowStream")).toBe("Arrow IPC stream"); + }); + + it("labels compression codecs", () => { + expect(compressionLabel("uncompressed")).toMatch(/uncompressed/i); + expect(compressionLabel("snappy")).toBe("Snappy"); + expect(compressionLabel("zstd")).toMatch(/zstd/i); + }); +}); + +describe("suggested export file names", () => { + it("maps each format to its extension", () => { + expect(columnarFormatExtension("parquet")).toBe("parquet"); + expect(columnarFormatExtension("arrowFile")).toBe("arrow"); + expect(columnarFormatExtension("arrowStream")).toBe("arrows"); + }); + + it("swaps an existing extension for the format's own", () => { + expect(suggestColumnarFileName("sales.csv", "parquet")).toBe("sales.parquet"); + expect(suggestColumnarFileName("sales.parquet", "arrowFile")).toBe("sales.arrow"); + expect(suggestColumnarFileName("no-ext", "arrowStream")).toBe("no-ext.arrows"); + }); + + it("does not treat a dotted directory as an extension boundary", () => { + expect(suggestColumnarFileName("v1.2/data", "parquet")).toBe("v1.2/data.parquet"); + }); +}); + +describe("columnar path routing", () => { + it("recognises every columnar extension, case-insensitively", () => { + expect(isColumnarPath("C:/x.parquet")).toBe(true); + expect(isColumnarPath("C:/x.ARROW")).toBe(true); + expect(isColumnarPath("C:/x.feather")).toBe(true); + expect(isColumnarPath("C:/x.ipc")).toBe(true); + expect(isColumnarPath("C:/x.arrows")).toBe(true); + }); + + it("leaves CSV and JSON alone", () => { + expect(isColumnarPath("C:/x.csv")).toBe(false); + expect(isColumnarPath("C:/x.json")).toBe(false); + }); +}); + +describe("defaults", () => { + it("open defaults preserve complex fields as JSON", () => { + expect(defaultColumnarOpenOptions()).toEqual({ + complexPolicy: "preserveJson", + fieldPolicies: {}, + cacheBudgetBytes: 0, + }); + }); + + it("export defaults match the Rust defaults (Snappy Parquet, typed)", () => { + expect(defaultColumnarExportOptions()).toEqual({ + format: "parquet", + compression: "snappy", + typed: true, + rowGroupRows: 0, + backup: "none", + }); + }); +}); + +describe("complex-field policy state", () => { + const base = defaultColumnarOpenOptions(); + + it("falls back to the default policy when no override is set", () => { + expect(effectivePolicy(base, "items")).toBe("preserveJson"); + expect(effectivePolicy({ complexPolicy: "reject" }, "items")).toBe("reject"); + expect(effectivePolicy({}, "items")).toBe("preserveJson"); + }); + + it("honours a per-field override over the default", () => { + const opts = setFieldPolicy(base, "items", "explode"); + expect(effectivePolicy(opts, "items")).toBe("explode"); + expect(effectivePolicy(opts, "other")).toBe("preserveJson"); + }); + + it("setFieldPolicy is immutable and merges with existing overrides", () => { + const a = setFieldPolicy(base, "items", "explode"); + const b = setFieldPolicy(a, "tags", "reject"); + expect(base.fieldPolicies).toEqual({}); + expect(a.fieldPolicies).toEqual({ items: "explode" }); + expect(b.fieldPolicies).toEqual({ items: "explode", tags: "reject" }); + }); + + it("collects the fields effectively set to explode, in order", () => { + const opts: ColumnarOpenOptions = { fieldPolicies: { b: "explode", a: "explode" } }; + expect(explodeFields(opts, ["a", "b", "c"])).toEqual(["a", "b"]); + }); +}); + +describe("open-mode plan", () => { + const insp = inspection({ complexFields: ["items", "tags"] }); + + it("keeps both modes available when nothing explodes", () => { + const plan = columnarOpenPlan(insp, defaultColumnarOpenOptions()); + expect(plan.exploded).toEqual([]); + expect(plan.requiresEditable).toBe(false); + expect(plan.indexedDisabledReason).toBeNull(); + expect(plan.errors).toEqual([]); + }); + + it("one explode forces editable and disables the indexed mode", () => { + const opts = setFieldPolicy(defaultColumnarOpenOptions(), "items", "explode"); + const plan = columnarOpenPlan(insp, opts); + expect(plan.exploded).toEqual(["items"]); + expect(plan.requiresEditable).toBe(true); + expect(plan.tooManyExplode).toBe(false); + expect(plan.indexedDisabledReason).toMatch(/editable/i); + expect(plan.errors).toEqual([]); + }); + + it("two explodes are a blocking error for both modes", () => { + let opts = setFieldPolicy(defaultColumnarOpenOptions(), "items", "explode"); + opts = setFieldPolicy(opts, "tags", "explode"); + const plan = columnarOpenPlan(insp, opts); + expect(plan.tooManyExplode).toBe(true); + expect(plan.errors).toHaveLength(1); + expect(plan.errors[0]).toMatch(/one field/i); + }); +}); + +describe("inspection display", () => { + it("labels the chunk unit per format and count", () => { + expect(chunkUnitLabel("parquet", 1)).toBe("row group"); + expect(chunkUnitLabel("parquet", 3)).toBe("row groups"); + expect(chunkUnitLabel("arrowFile", 1)).toBe("record batch"); + expect(chunkUnitLabel("arrowStream", 2)).toBe("record batches"); + }); + + it("formats the editable-memory estimate", () => { + expect(estimatedMemoryLabel(inspection({ estimatedMemory: 5 * 1024 * 1024 }))).toBe("5.0 MB"); + }); + + it("computes nesting depth and leaf name of a flattened path", () => { + expect(columnDepth("id")).toBe(0); + expect(columnDepth("addr.city")).toBe(1); + expect(columnDepth("a.b.c")).toBe(2); + expect(leafName("addr.city")).toBe("city"); + // An escaped dot is part of the leaf, not a separator. + expect(columnDepth("weird\\.name")).toBe(0); + expect(leafName("weird\\.name")).toBe("weird.name"); + }); +}); diff --git a/src/lib/columnar.ts b/src/lib/columnar.ts new file mode 100644 index 0000000..3e06a7d --- /dev/null +++ b/src/lib/columnar.ts @@ -0,0 +1,173 @@ +// Pure helpers for the Parquet / Arrow interop UI (F32): format/compression +// labels, suggested export names, complex-field policy state, and the +// open-mode plan (which policies force an editable open, which are outright +// invalid). The policy/mode rules mirror the read side exactly so the +// ParquetInspectDialog surfaces the same constraints the backend enforces — +// immediately and offline. No I/O here; everything is testable in isolation. + +import { splitJsonPath } from "./jsonImport"; +import { formatBytes } from "./save"; +import type { + ColumnarCompression, + ColumnarExportOptions, + ColumnarFormat, + ColumnarInspection, + ColumnarOpenOptions, + ComplexPolicy, +} from "../types"; + +// ----- formats & compression ------------------------------------------------ + +const FORMAT_LABELS: Record = { + parquet: "Apache Parquet", + // Feather v2 IS the Arrow IPC file container — keep the alias visible. + arrowFile: "Arrow IPC file (Feather v2)", + arrowStream: "Arrow IPC stream", +}; + +export function columnarFormatLabel(format: ColumnarFormat): string { + return FORMAT_LABELS[format]; +} + +const COMPRESSION_LABELS: Record = { + uncompressed: "None (uncompressed)", + snappy: "Snappy", + zstd: "Zstandard (zstd)", +}; + +export function compressionLabel(compression: ColumnarCompression): string { + return COMPRESSION_LABELS[compression]; +} + +/** File extension (no dot) the backend writes for each container. */ +export function columnarFormatExtension(format: ColumnarFormat): string { + switch (format) { + case "parquet": + return "parquet"; + case "arrowFile": + return "arrow"; + case "arrowStream": + return "arrows"; + } +} + +/** Replace a name's extension (or append) so it matches the chosen format. */ +export function suggestColumnarFileName(base: string, format: ColumnarFormat): string { + const ext = columnarFormatExtension(format); + const stem = base.replace(/\.[^.\\/]+$/, ""); + return `${stem}.${ext}`; +} + +/** File extensions that route through the columnar inspect/open pipeline. */ +export const COLUMNAR_EXTENSIONS = ["parquet", "arrow", "feather", "ipc", "arrows"] as const; + +/** Whether a path should open through the F32 inspect dialog. */ +export function isColumnarPath(path: string): boolean { + const lower = path.toLowerCase(); + return COLUMNAR_EXTENSIONS.some((ext) => lower.endsWith(`.${ext}`)); +} + +// ----- defaults ------------------------------------------------------------- + +/** Fresh open options: complex fields preserved as JSON, backend cache budget. */ +export function defaultColumnarOpenOptions(): ColumnarOpenOptions { + return { complexPolicy: "preserveJson", fieldPolicies: {}, cacheBudgetBytes: 0 }; +} + +/** Fresh export options, matching the Rust `ColumnarExportOptions` defaults. */ +export function defaultColumnarExportOptions(): ColumnarExportOptions { + return { + format: "parquet", + compression: "snappy", + typed: true, + rowGroupRows: 0, + backup: "none", + }; +} + +// ----- complex-field policy state ------------------------------------------- + +/** The policy in effect for one complex field: its override, else the default. */ +export function effectivePolicy(options: ColumnarOpenOptions, path: string): ComplexPolicy { + return options.fieldPolicies?.[path] ?? options.complexPolicy ?? "preserveJson"; +} + +/** Return new options with `path` overridden to `policy` (immutably). */ +export function setFieldPolicy( + options: ColumnarOpenOptions, + path: string, + policy: ComplexPolicy, +): ColumnarOpenOptions { + return { + ...options, + fieldPolicies: { ...(options.fieldPolicies ?? {}), [path]: policy }, + }; +} + +/** The complex fields whose effective policy is `explode`, in the given order. */ +export function explodeFields(options: ColumnarOpenOptions, complexFields: string[]): string[] { + return complexFields.filter((path) => effectivePolicy(options, path) === "explode"); +} + +// ----- open-mode plan ------------------------------------------------------- + +/** + * The consequences of the current policy selection for the two open modes. + * Mirrors the read side: exploding a list changes the row count, so it is + * editable-open only and at most ONE column may explode per open. + */ +export interface ColumnarOpenPlan { + /** Complex fields currently set to explode. */ + exploded: string[]; + /** More than one explode selected — invalid for BOTH modes. */ + tooManyExplode: boolean; + /** Any explode selected — the indexed (read-only) mode cannot represent it. */ + requiresEditable: boolean; + /** Why the indexed button is unavailable, or null when it is available. */ + indexedDisabledReason: string | null; + /** Blocking errors that make either open invalid. */ + errors: string[]; +} + +export function columnarOpenPlan( + inspection: ColumnarInspection, + options: ColumnarOpenOptions, +): ColumnarOpenPlan { + const exploded = explodeFields(options, inspection.complexFields); + const tooManyExplode = exploded.length > 1; + const requiresEditable = exploded.length > 0; + const errors: string[] = []; + if (tooManyExplode) { + errors.push( + `Only one field can be exploded into rows per open (${exploded.length} selected) — set the others to keep-as-JSON or drop.`, + ); + } + const indexedDisabledReason = requiresEditable + ? "Exploding a list multiplies the row count, which a read-only index can't represent — convert to editable instead." + : null; + return { exploded, tooManyExplode, requiresEditable, indexedDisabledReason, errors }; +} + +// ----- inspection display --------------------------------------------------- + +/** Human label for the chunk unit: Parquet row groups vs Arrow record batches. */ +export function chunkUnitLabel(format: ColumnarFormat, count: number): string { + if (format === "parquet") return count === 1 ? "row group" : "row groups"; + return count === 1 ? "record batch" : "record batches"; +} + +/** "12.3 MB"-style label for the editable-memory estimate. */ +export function estimatedMemoryLabel(inspection: ColumnarInspection): string { + return formatBytes(inspection.estimatedMemory); +} + +/** Nesting depth of a flattened path (0 = top level), for indented rendering. */ +export function columnDepth(name: string): number { + return splitJsonPath(name).length - 1; +} + +/** The last (leaf) segment of a flattened path, for the nested tree display. */ +export function leafName(name: string): string { + const segments = splitJsonPath(name); + return segments[segments.length - 1] ?? name; +} diff --git a/src/lib/commandDefs.ts b/src/lib/commandDefs.ts index 1ffdb49..8b52ef5 100644 --- a/src/lib/commandDefs.ts +++ b/src/lib/commandDefs.ts @@ -72,6 +72,14 @@ function staticCommands(): AppCommand[] { allowInEditable: true, run: () => void state().openJsonDialog(), }, + { + id: "file.openColumnar", + title: "Open Parquet/Arrow…", + keywords: ["parquet", "arrow", "feather", "ipc", "columnar", "import", "typed"], + category: "File", + allowInEditable: true, + run: () => void state().openColumnarDialog(), + }, { id: "file.save", title: "Save", @@ -108,6 +116,14 @@ function staticCommands(): AppCommand[] { unavailableReason: needsDoc, run: () => openModal("jsonExport"), }, + { + id: "file.exportColumnar", + title: "Export as Parquet/Arrow…", + keywords: ["parquet", "arrow", "feather", "ipc", "columnar", "typed", "snappy", "zstd"], + category: "Export", + unavailableReason: needsDoc, + run: () => openModal("columnarExport"), + }, { id: "file.closeTab", title: "Close tab", diff --git a/src/lib/project.test.ts b/src/lib/project.test.ts index d7c6871..8a3c15b 100644 --- a/src/lib/project.test.ts +++ b/src/lib/project.test.ts @@ -32,6 +32,7 @@ import { projectDirty, projectSnapshot, projectSnapshotsEqual, + restoreOpenRoute, setCaseSensitivePaths, statusDisplay, type PanelLayout, @@ -119,6 +120,25 @@ describe("pathKey", () => { }); }); +describe("restoreOpenRoute", () => { + it("routes columnar sources to a non-interactive indexed open", () => { + expect(restoreOpenRoute("C:\\data\\sales.parquet")).toBe("columnarIndexed"); + expect(restoreOpenRoute("/home/u/events.arrow")).toBe("columnarIndexed"); + expect(restoreOpenRoute("C:/data/EVENTS.FEATHER")).toBe("columnarIndexed"); + expect(restoreOpenRoute("C:/data/stream.arrows")).toBe("columnarIndexed"); + expect(restoreOpenRoute("C:/data/legacy.ipc")).toBe("columnarIndexed"); + }); + + it("routes every other source through the ordinary open pipeline", () => { + expect(restoreOpenRoute("C:\\data\\a.csv")).toBe("standard"); + expect(restoreOpenRoute("C:\\data\\a.json")).toBe("standard"); + expect(restoreOpenRoute("C:\\data\\a.jsonl")).toBe("standard"); + expect(restoreOpenRoute("C:\\data\\a.tsv")).toBe("standard"); + expect(restoreOpenRoute("C:\\data\\archive.zip")).toBe("standard"); + expect(restoreOpenRoute("C:\\data\\log.txt.gz")).toBe("standard"); + }); +}); + describe("statusDisplay", () => { const cases: [SourceStatus, "ok" | "warn" | "error"][] = [ ["ok", "ok"], diff --git a/src/lib/project.ts b/src/lib/project.ts index 4b8f5db..635f3d1 100644 --- a/src/lib/project.ts +++ b/src/lib/project.ts @@ -3,6 +3,7 @@ // can be unit-tested without a backend — status mapping, per-source resolution // building, source/tab section capture, and project dirty-state derivation. +import { isColumnarPath } from "./columnar"; import type { AnnotationsExport, DocumentMeta, @@ -555,6 +556,27 @@ export function buildAnnotationsSection( return out; } +// ----- restore open routing --------------------------------------------------- + +/** + * How a project restore should open one source's file. `columnarIndexed` opens + * a Parquet / Arrow source directly as an indexed read-only document; `standard` + * uses the ordinary open pipeline (CSV, JSON, archives, …). + */ +export type RestoreOpenRoute = "columnarIndexed" | "standard"; + +/** + * Decide how a project restore should open one source (pure). A columnar + * (Parquet / Arrow) path must NOT go through the interactive inspect dialog the + * normal open uses: a restore is non-interactive, so it reopens the source + * directly as an indexed read-only document instead of leaving the tab + * un-created until the user manually confirms a dialog. Every other path + * restores through the ordinary open pipeline. + */ +export function restoreOpenRoute(path: string): RestoreOpenRoute { + return isColumnarPath(path) ? "columnarIndexed" : "standard"; +} + /** * Order the given tab ids to match a plan's tab order: tabs whose path matches * a plan entry come first (in plan order), any others keep their relative order diff --git a/src/lib/tauri.ts b/src/lib/tauri.ts index a1c9133..3745167 100644 --- a/src/lib/tauri.ts +++ b/src/lib/tauri.ts @@ -55,6 +55,10 @@ import type { JsonExportOptions, JsonImportOptions, JsonImportPreview, + ColumnarInspection, + ColumnarOpenOptions, + ColumnarExportOptions, + ColumnarExportReport, FileFingerprint, FileProfile, FilterGroup, @@ -1154,6 +1158,58 @@ export const jsonExport = ( expectedRevision: number, ) => invoke("json_export", { docId, path, options, scope, expectedRevision }); +// ----- Parquet / Arrow interop (F32) --------------------------------------- + +/** + * Inspect a Parquet / Arrow IPC file BEFORE any open: container format, row + * and row-group/batch counts, columns mapped to the F31 logical types, + * compression codecs, nested (complex) fields, and the editable-memory + * estimate with its decision flag. Reads metadata only; opens nothing. + */ +export const columnarInspect = (path: string) => + invoke("columnar_inspect", { path }); + +/** + * Open a Parquet / Arrow file as an indexed READ-ONLY document (windowed + * columnar reads behind the same grid/filter/export machinery as an F10 + * indexed CSV). Runs under the "openIndexed" job kind, so the existing + * completion path adds the tab; the document registers under the returned + * docId when the job finishes. Explode policies are rejected here. + */ +export const columnarOpenIndexed = (path: string, options?: ColumnarOpenOptions) => + invoke("columnar_open_indexed", { path, options }); + +/** + * Open a Parquet / Arrow file straight into a fully editable in-memory + * document, honouring every complex-field policy including exploding one list + * column into rows. Re-runs the memory estimate first; pass `force` after an + * explicit user decision. Opens UNSAVED with no path (Save must not overwrite + * the binary source with CSV bytes). + */ +export const columnarOpenEditable = (path: string, options: ColumnarOpenOptions, force: boolean) => + invoke("columnar_open_editable", { path, options, force }); + +/** + * Start a scoped, typed export to Parquet (uncompressed/Snappy/Zstd) or Arrow + * IPC file/stream as a cancellable "export" job. The scope resolves and the + * revision is checked BEFORE the job spawns (the invoke rejects); everything + * streams through the atomic-save pipeline, so failure/cancel removes the + * staging file and never touches an existing destination. Fetch the outcome + * with {@link getColumnarExportReport} after the `job-finished` event. + */ +export const columnarExport = ( + docId: number, + path: string, + options: ColumnarExportOptions, + scope: ExportScope, + expectedRevision: number, +) => invoke("columnar_export", { docId, path, options, scope, expectedRevision }); + +/** The report of a finished columnar export (rows, bytes, per-column + * invalid-cell counts), by its job id. */ +export const getColumnarExportReport = (jobId: number) => + invoke("get_columnar_export_report", { jobId }); + // ----- project workspaces (F37) --------------------------------------------- // The ProjectStore is THE persistence boundary: typed, versioned sections are // written through `project_set_section` and flushed atomically by `project_save`. diff --git a/src/store/useStore.ts b/src/store/useStore.ts index 638f2f2..b67a61a 100644 --- a/src/store/useStore.ts +++ b/src/store/useStore.ts @@ -22,6 +22,11 @@ import { annotationExportName } from "../lib/annotations"; import { currentOpenOptions, fingerprintKey } from "../lib/reopen"; import { defaultImportOptions } from "../lib/jsonImport"; import { suggestJsonFileName } from "../lib/jsonExport"; +import { + defaultColumnarOpenOptions, + isColumnarPath, + suggestColumnarFileName, +} from "../lib/columnar"; import { isLegacyEncoding } from "../lib/save"; import { availableOnlyChoices, @@ -38,6 +43,7 @@ import { pathKey, projectDirty, projectSnapshot, + restoreOpenRoute, type PanelLayout, type ProjectSnapshot, type SourceAnnotationsSection, @@ -58,6 +64,10 @@ import type { JsonExportOptions, JsonImportOptions, JsonImportPreview, + ColumnarInspection, + ColumnarOpenOptions, + ColumnarExportOptions, + ColumnarExportReport, FileProfile, FilterGroup, FindMatch, @@ -172,6 +182,7 @@ export type ModalName = | "schema" | "dictionary" | "jsonExport" + | "columnarExport" | "sampling" | "tagToColumn" | "annotationExport" @@ -181,10 +192,19 @@ export type ModalName = const FILE_FILTERS = [ { name: "Delimited text", extensions: ["csv", "tsv", "tab", "txt", "psv", "dat"] }, { name: "JSON (F33)", extensions: ["json", "jsonl", "ndjson"] }, + { name: "Columnar (F32)", extensions: ["parquet", "arrow", "feather", "ipc", "arrows"] }, { name: "Compressed (F17)", extensions: ["gz", "zip"] }, { name: "All files", extensions: ["*"] }, ]; +/** File filters for a Parquet / Arrow export target (F32). */ +const COLUMNAR_FILE_FILTERS = [ + { name: "Apache Parquet", extensions: ["parquet"] }, + { name: "Arrow IPC file (Feather v2)", extensions: ["arrow", "feather"] }, + { name: "Arrow IPC stream", extensions: ["arrows", "ipc"] }, + { name: "All files", extensions: ["*"] }, +]; + /** File filters for a JSON / JSON Lines export target (F33). */ const JSON_FILE_FILTERS = [ { name: "JSON", extensions: ["json"] }, @@ -475,6 +495,9 @@ export interface IndexingState { /** Bytes scanned (open/reindex/extract) or rows materialised (convert). */ processed: number; total: number | null; + /** Unit of `processed`/`total` for progress display. CSV opens scan bytes; + * columnar (F32) opens count rows. Absent = bytes. */ + unit?: "bytes" | "rows"; /** Extraction bookkeeping (F17). */ archiveToken?: number; archiveEntry?: string | null; @@ -597,6 +620,32 @@ export interface JsonImportState { scanError: string | null; } +/** + * Parquet / Arrow open flow state (F32). Non-null while the inspect dialog is + * open. The inspection is fetched once (metadata only); the actual open then + * runs as an "openIndexed" job whose completion adds the tab through the shared + * indexing pipeline, so this slice clears the moment an open starts. + */ +export interface ColumnarOpenState { + path: string; + fileName: string; + /** The pre-open inspection, or null while it is still loading. */ + inspection: ColumnarInspection | null; + loading: boolean; + error: string | null; +} + +/** + * Result of the most recent Parquet / Arrow export (F32). Drives the export + * dialog's running/done phases; `report` carries the per-column invalid-cell + * counts the typed-export warning surface shows. + */ +export interface ColumnarExportState { + running: boolean; + report: ColumnarExportReport | null; + error: string | null; +} + /** A running sampling/partitioning job (F48). Unlike a derive job it can emit * MANY new documents (one per partition) or none at all (a direct export). */ export interface SampleState { @@ -899,6 +948,10 @@ interface Store { deriveError: string | null; /** JSON / JSON Lines import flow (F33); non-null while its dialog is open. */ jsonImport: JsonImportState | null; + /** Parquet / Arrow inspect+open flow (F32); non-null while its dialog is open. */ + columnarOpen: ColumnarOpenState | null; + /** Result of the most recent Parquet / Arrow export (F32). */ + columnarExportResult: ColumnarExportState; /** Running sampling/partitioning job (F48), if any. */ sample: SampleState | null; /** Error from the last sampling job, for the dialog that started it. */ @@ -1016,6 +1069,8 @@ interface Store { openDialog: () => Promise; /** File picker filtered to JSON / JSON Lines, routed through the open flow. */ openJsonDialog: () => Promise; + /** File picker filtered to Parquet / Arrow, routed through the open flow (F32). */ + openColumnarDialog: () => Promise; openPath: (path: string) => Promise; newDoc: () => Promise; closeTab: (id: number) => Promise; @@ -1352,6 +1407,25 @@ interface Store { /** Prompt for a path and export the active document as JSON (F33). */ exportJson: (options: JsonExportOptions, scope: ExportScope) => Promise; + // Parquet / Arrow interop (F32) + /** Inspect a columnar file and open the inspect dialog. */ + openColumnarInspect: (path: string) => Promise; + /** Restore a columnar source non-interactively (F37): reopen it directly as + * an indexed read-only document, awaiting the open job, WITHOUT the inspect + * dialog. Used by project open so a Parquet / Arrow tab is actually created. */ + openColumnarRestore: (path: string) => Promise; + /** Close the inspect dialog. */ + dismissColumnarInspect: () => void; + /** Open the inspected file as an indexed READ-ONLY document. */ + columnarOpenIndexed: (options: ColumnarOpenOptions) => Promise; + /** Open the inspected file fully editable in memory (`force` past the + * memory guard after an explicit user decision). */ + columnarOpenEditable: (options: ColumnarOpenOptions, force: boolean) => Promise; + /** Prompt for a path and export the active document to Parquet / Arrow (F32). */ + runColumnarExport: (options: ColumnarExportOptions, scope: ExportScope) => Promise; + /** Reset the columnar export result (dialog closed / reopened). */ + clearColumnarExport: () => void; + // data-cleaning transforms (F06) /** * Apply a previewed transform (one undo step). Returns whether it was @@ -2211,7 +2285,16 @@ export const useStore = create((set, get) => { return; } set({ projectOpenPending: { plan: pending.plan, remaining: step.remaining } }); - await get().openPath(step.path); + // A restore is non-interactive: a columnar (Parquet / Arrow) source + // must NOT route through the interactive inspect dialog `openPath` + // would open — that returns without creating a tab, leaving the + // restored source missing until the user manually confirmed it. Reopen + // it directly as an indexed read-only document instead (F37). + if (restoreOpenRoute(step.path) === "columnarIndexed") { + await get().openColumnarRestore(step.path); + } else { + await get().openPath(step.path); + } } } finally { pumpingProjectOpen = false; @@ -2293,6 +2376,8 @@ export const useStore = create((set, get) => { derive: null, deriveError: null, jsonImport: null, + columnarOpen: null, + columnarExportResult: { running: false, report: null, error: null }, sample: null, sampleError: null, samplingInitialMode: "sampling", @@ -2724,6 +2809,11 @@ export const useStore = create((set, get) => { if (typeof selected === "string") await get().openPath(selected); }, + openColumnarDialog: async () => { + const selected = await openFileDialog({ multiple: false, filters: COLUMNAR_FILE_FILTERS }); + if (typeof selected === "string") await get().openPath(selected); + }, + openPath: async (path) => { const existing = get().tabs.find((t) => t.path === path); if (existing) { @@ -2758,6 +2848,12 @@ export const useStore = create((set, get) => { await get().openJsonImport(path); return; } + // F32: Parquet / Arrow IPC route through the inspect dialog (metadata, + // schema, open-indexed vs convert-editable choice). + if (isColumnarPath(path)) { + await get().openColumnarInspect(path); + return; + } set({ busy: true, error: null }); try { // F10: estimate the in-memory cost first. Large files pause here and @@ -5698,6 +5794,169 @@ export const useStore = create((set, get) => { } }, + // ----- Parquet / Arrow interop (F32) ------------------------------------------ + + openColumnarInspect: async (path) => { + // Reuse an already-open tab for this source rather than re-inspecting. + const existing = get().tabs.find((t) => t.path === path); + if (existing) { + set((s) => switchPatch(s, existing.id)); + return; + } + const fileName = path.split(/[\\/]/).pop() ?? path; + set({ columnarOpen: { path, fileName, inspection: null, loading: true, error: null } }); + try { + const inspection = await api.columnarInspect(path); + // The dialog may have been dismissed while the invoke was in flight. + if (get().columnarOpen?.path !== path) return; + set((s) => + s.columnarOpen ? { columnarOpen: { ...s.columnarOpen, inspection, loading: false } } : {}, + ); + } catch (e) { + if (get().columnarOpen?.path !== path) return; + set((s) => + s.columnarOpen + ? { columnarOpen: { ...s.columnarOpen, loading: false, error: String(e) } } + : {}, + ); + } + }, + + openColumnarRestore: async (path) => { + // Non-interactive columnar restore (F37 project open). Reuse an already + // open tab, otherwise reopen the source as an indexed READ-ONLY document + // with default policies — no inspect dialog. Indexed is the memory-safe, + // prompt-free reproduction of a columnar source (it never hits the F10 + // memory decision, and indexed documents stay read-only unless the user + // later converts them). The open is a job, so await it here rather than + // via the shared `indexing` pipeline: the restore's later steps (tab + // order, annotations, view reapply) need the tab to exist synchronously. + const existing = get().tabs.find((t) => t.path === path); + if (existing) { + set((s) => switchPatch(s, existing.id)); + return; + } + try { + const started = await api.columnarOpenIndexed(path, defaultColumnarOpenOptions()); + const finished = await awaitJob(started.jobId); + if (finished.status !== "done") { + // A restore only warns on incompatibility; a failed columnar open + // surfaces its message but never blocks the rest of the project. + if (finished.status === "failed") { + set({ error: finished.error ?? `Could not restore ${path}` }); + } + return; + } + const meta = await api.getMeta(started.docId); + set((s) => ({ ...switchPatch(s, meta.id), tabs: [...s.tabs, meta] })); + pushRecent(path); + } catch (e) { + set({ error: String(e) }); + } + }, + + dismissColumnarInspect: () => set({ columnarOpen: null }), + + columnarOpenIndexed: async (options) => { + const st = get().columnarOpen; + if (!st || get().indexing) return; + // The open job reports progress in rows (ctx.set_total = row count); use + // the inspected row count as the placeholder until the first event. + const total = st.inspection?.rowCount ?? null; + try { + const started = await api.columnarOpenIndexed(st.path, options); + // Hand off to the shared indexing pipeline; its completion adds the tab. + set({ + columnarOpen: null, + indexing: { + jobId: started.jobId, + docId: started.docId, + kind: "openIndexed", + path: st.path, + processed: 0, + total, + unit: "rows", + }, + }); + consumeEarlyFinish(started.jobId); + } catch (e) { + // Keep the dialog open and surface the failure inline. + set((s) => + s.columnarOpen + ? { columnarOpen: { ...s.columnarOpen, error: String(e) } } + : { error: String(e) }, + ); + } + }, + + columnarOpenEditable: async (options, force) => { + const st = get().columnarOpen; + if (!st || get().indexing) return; + // Reports progress in rows, like the indexed open above. + const total = st.inspection?.rowCount ?? null; + try { + const started = await api.columnarOpenEditable(st.path, options, force); + set({ + columnarOpen: null, + indexing: { + jobId: started.jobId, + docId: started.docId, + kind: "openIndexed", + path: st.path, + processed: 0, + total, + unit: "rows", + }, + }); + consumeEarlyFinish(started.jobId); + } catch (e) { + // Typically the memory-estimate refusal (force=false); keep the dialog + // open so the user can choose "open editable anyway". + set((s) => + s.columnarOpen + ? { columnarOpen: { ...s.columnarOpen, error: String(e) } } + : { error: String(e) }, + ); + } + }, + + runColumnarExport: async (options, scope) => { + const meta = activeMeta(); + if (!meta || get().columnarExportResult.running) return; + const chosen = await saveFileDialog({ + defaultPath: suggestColumnarFileName(meta.fileName, options.format), + filters: COLUMNAR_FILE_FILTERS, + }); + if (!chosen) return; + set({ columnarExportResult: { running: true, report: null, error: null } }); + try { + // The invoke resolves the scope and checks the revision up front, so an + // invalid scope / stale snapshot rejects here before any job spawns. + const jobId = await api.columnarExport(meta.id, chosen, options, scope, meta.revision); + const finished = await awaitJob(jobId); + if (finished.status === "done") { + const report = await api.getColumnarExportReport(jobId).catch(() => null); + set({ columnarExportResult: { running: false, report, error: null } }); + } else if (finished.status === "failed") { + set({ + columnarExportResult: { + running: false, + report: null, + error: finished.error ?? "export failed", + }, + }); + } else { + // Cancelled: back to the config phase with nothing written. + set({ columnarExportResult: { running: false, report: null, error: null } }); + } + } catch (e) { + set({ columnarExportResult: { running: false, report: null, error: String(e) } }); + } + }, + + clearColumnarExport: () => + set({ columnarExportResult: { running: false, report: null, error: null } }), + // ----- compare (F09) ----------------------------------------------------------- runCompare: async (rightDocId, spec) => { diff --git a/src/types.ts b/src/types.ts index 3da69c6..d5344d8 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1868,6 +1868,113 @@ export interface JsonExportOptions { backup: BackupPolicy; } +// ----- Parquet / Arrow interop (F32) ---------------------------------------- + +/** + * The three supported columnar container formats (mirrors Rust + * `ColumnarFormat`). `arrowFile` is Apache Arrow IPC file, a.k.a. Feather v2 — + * the same container; surface that alias in UI copy. + */ +export type ColumnarFormat = "parquet" | "arrowFile" | "arrowStream"; + +/** + * What to do with a nested field the flat text contract cannot carry directly + * (list, map, …) — mirrors Rust `ComplexPolicy`. `explode` is editable-open + * only and permits at most one list column per open. + */ +export type ComplexPolicy = "preserveJson" | "explode" | "reject"; + +/** Options for inspect / open-indexed / open-editable (mirrors Rust + * `ColumnarOpenOptions`). All fields are optional; the backend defaults apply. */ +export interface ColumnarOpenOptions { + /** Default policy for every complex field. */ + complexPolicy?: ComplexPolicy; + /** Per-field overrides, keyed by the flattened path-based column name. */ + fieldPolicies?: Record; + /** Decoded-block LRU budget override in bytes (0 = backend default). */ + cacheBudgetBytes?: number; +} + +/** One column as reported by `columnar_inspect` (mirrors Rust + * `InspectedColumn`). */ +export interface InspectedColumn { + /** Flattened path-based name (struct segments escaped and dot-joined). */ + name: string; + /** The arrow type for display (`Int64`, `Timestamp(µs, Europe/Berlin)`, …). */ + arrowType: string; + /** The F31 logical type the column maps to. */ + logicalType: LogicalType; + nullable: boolean; + /** Preserved IANA timezone for zoned timestamp columns. */ + timeZone?: string; + /** Whether the column came out of a nested field (struct flattening or a + * complex-field policy). */ + nested: boolean; +} + +/** Everything the open dialog needs BEFORE opening a columnar file (mirrors + * Rust `ColumnarInspection`). */ +export interface ColumnarInspection { + /** Wire format name: `parquet` / `arrowFile` / `arrowStream`. */ + format: ColumnarFormat; + rowCount: number; + /** Parquet row groups / Arrow record batches. */ + chunkCount: number; + /** Distinct parquet compression codecs; absent for Arrow IPC. */ + compression?: string; + /** Columns under the DEFAULT policies (complex fields preserved as JSON). */ + columns: InspectedColumn[]; + /** Flattened paths of complex fields that take a `ComplexPolicy`. */ + complexFields: string[]; + /** Rough bytes the fully editable in-memory document would need. */ + estimatedMemory: number; + /** Whether opening editable should require an explicit choice. */ + needsDecision: boolean; + fileSize: number; +} + +/** Parquet compression codec choice (mirrors Rust `ColumnarCompression`). + * Applies to Parquet only; Arrow IPC output is always uncompressed. */ +export type ColumnarCompression = "uncompressed" | "snappy" | "zstd"; + +/** Options for a columnar export (mirrors Rust `ColumnarExportOptions`). */ +export interface ColumnarExportOptions { + /** Output container: Parquet, Arrow IPC file (Feather v2) or stream. */ + format: ColumnarFormat; + /** Parquet compression codec (ignored for the Arrow IPC formats). */ + compression: ColumnarCompression; + /** Emit typed arrow columns for columns with a declared F31 schema; + * `false` writes every column as Utf8 text. */ + typed: boolean; + /** Parquet only: max rows per row group (0 = writer default). Smaller + * groups give the read side's statistics pruning more to skip. */ + rowGroupRows: number; + /** Backup policy for the previous destination file. */ + backup: BackupPolicy; +} + +/** Per-column invalid-cell total on a finished export (mirrors Rust + * `ColumnWarning`). */ +export interface ColumnWarning { + name: string; + /** Cells written as NULL because they could not be represented under the + * declared schema / chosen arrow type. */ + invalidCells: number; +} + +/** What a finished columnar export produced (mirrors Rust + * `ColumnarExportReport`). */ +export interface ColumnarExportReport { + format: ColumnarFormat; + rows: number; + columns: number; + bytes: number; + /** Total cells exported as NULL with a warning. */ + invalidCells: number; + /** Per-column breakdown (only columns with at least one warning). */ + columnWarnings: ColumnWarning[]; +} + // ----- project workspaces (F37) --------------------------------------------- /** Header state of the open project, for the project bar (F37). */