feat(f32): Parquet and Arrow interoperability - #46
Conversation
Read engine for Parquet / Arrow IPC file (Feather v2) / Arrow IPC stream:
- arrow 59 (ipc) + parquet 59 (arrow, snap, zstd) locked deps.
- parquet_arrow.rs: format sniffing; inspection (rows, F31-mapped columns,
row-group/batch count, codecs, nested fields, editable-memory estimate);
ColumnarHandle — an indexed read-only backing with windowed reads over
row groups / record batches and a bounded LRU of decoded text blocks.
- Typed-value -> canonical text: i64/u64 as decimal strings (never through
JS numbers), exact decimal rendering from mantissa+scale, shortest-float
display, true/false, ISO dates, naive/zoned timestamps (zone metadata on
ColumnSchema.timeZone; zoned values render as the UTC instant), binary as
hex. Columnar NULL is None in the tabular Option plane, Some("") stays an
empty string; the grid text plane renders NULL as an empty cell.
- Nested: structs always flatten to escaped path-based names; lists/maps
take an explicit policy (preserveJson / reject / explode-on-editable-open,
single list column). Editable materialisation preserves null-vs-empty via
collision-free per-column null tokens recorded in the schema.
- Document integration: Backing::Columnar wired through visit_rows /
visit_at so the grid, filters, sorts and export work unchanged (backing
reports indexedReadOnly on the wire on purpose); DocumentSource reads the
Option plane for columnar docs; convert-to-editable and reindex commands
branch for columnar documents (own memory estimate, token-preserving
conversion, reindex refused).
- Filter acceleration: equality + range conditions on numeric/date columns
prune parquet row groups via their statistics (conservative: only when
typed bounds prove no match under an unchanged schema); matching_rows
visits only surviving ranges and returns results identical to a full
scan (tested).
- 16 unit tests: type matrix incl. i64::MIN/MAX, u64::MAX and decimal
scale; null-vs-empty distinctness; tz metadata round-trip; flatten
determinism + dot escaping; list/map policy matrix; explode semantics;
null-token collision escalation; IPC file/stream equivalence; dictionary
decoding; stats pruning correctness + graceful fallbacks; document
integration; LRU budget; cancellation leaves no artifacts; inspection.
Export side, commands/UI and typed parquet/arrow writing land in the
follow-up F32 stages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Write side of the Parquet & Arrow interop plus the F32 command surface. - src-tauri/src/columnar_export.rs (new): scoped typed export engine. Any export scope streams to Parquet (uncompressed/Snappy/Zstd, optional row-group size bound) or Arrow IPC file/stream through the atomic-save pipeline as a cancellable job. Declared F31 logical types map to arrow types: Integer -> Int64, or UInt64 when a typing pre-pass finds values beyond i64::MAX and none negative (u64::MAX round-trips losslessly); Decimal -> Decimal128 with exactly unified precision/scale; Float -> Float64; Boolean; Date -> Date32; Datetime -> Timestamp us (ns when sub-microsecond digits exist) carrying ColumnSchema.time_zone as the arrow timezone; Text/Uuid/Json -> Utf8 verbatim. Rows read through the Option plane so columnar NULL stays distinct from the empty string; schema null tokens and empty typed cells export as null without warning; unparseable/unrepresentable cells export as null with per-column counted warnings on a job-keyed report. - commands.rs: columnar_inspect, columnar_open_indexed (job-registered read-only document), columnar_open_editable (memory check + force, explode-capable, opens unsaved with no path so Save cannot clobber the binary source), columnar_export (revision-guarded, fail-fast scope validation), get_columnar_export_report. Converting an open columnar document to editable now also detaches path/fingerprint and marks it derived-unsaved for the same reason. - parquet_arrow.rs: ColumnarFormat is now Deserialize so export options share the inspection wire names. - 21 new tests: null-vs-empty distinctness both directions, i64/u64 extremes, warning policy, decimal scale unification and exact columnar round trip, zoned timestamp metadata + instants, ns unit selection, untyped/typed-false verbatim text, scope exactness, empty scope, all three containers, compression codecs, row-group bound + stats-pruned filtered reads equal to full scan, cancel cleanup, stale revision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Frontend for the F32 Parquet/Arrow interop backend (parts 1-2): inspect + open dialog, scoped columnar export dialog, open-pipeline wiring, and the TS/store surface. - types.ts: mirror the Rust DTOs (ColumnarFormat, ComplexPolicy, ColumnarOpenOptions, InspectedColumn, ColumnarInspection, ColumnarCompression, ColumnarExportOptions, ColumnWarning, ColumnarExportReport). - tauri.ts: columnarInspect / columnarOpenIndexed / columnarOpenEditable / columnarExport / getColumnarExportReport wrappers. - lib/columnar.ts (+ test): pure logic — format/compression labels, suggested export names, complex-field policy state, and the open-mode plan (explode forces editable, one field max). 19 vitest cases. - store: columnarOpen inspect-flow slice and columnarExportResult slice with actions; openPath routes .parquet/.arrow/.feather/.ipc/.arrows through the inspect dialog; columnar opens hand off to the shared "openIndexed" job so the existing completion path adds the tab. Progress reports rows (new IndexingState.unit) so the open-mode modal labels them correctly. - ParquetInspectDialog: format/counts/codec/memory summary, F31-mapped schema table with nested indentation and timezones, per-field policy pickers, indexed vs convert-editable choice behind a memory warning. - ColumnarExportDialog: format + Parquet compression/row-group-size + typed toggle + scope, with a done phase surfacing per-column invalid-cell counts. - commandDefs: "Open Parquet/Arrow…" and "Export as Parquet/Arrow…". - CHANGELOG + README entries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Consolidate the three staged F32 CHANGELOG entries (read engine, typed export & commands, UI) into a single user-facing "Parquet & Arrow interoperability" entry under [Unreleased]. README bullets already fit their Features groups. All gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebase feat/f32-parquet onto feat/f42-highlight (80dfd60). The new f42 commit (codex review — cross-column projection, stale overlays, clear-all draft) touched only src-tauri/src/highlight.rs, src/components/Grid.tsx, and src/components/HighlightRulesDialog.tsx — disjoint from every file f32 changes, so the replay was conflict-free. Full gates green on the first compile against the chain (npm lint/format/typecheck/test 392, build; cargo fmt/clippy; cargo test --lib 728, up from 726 as the rebase pulled in the two new highlight.rs unit tests) — no source fixes required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c9f53f82bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if !force { | ||
| let probe = path.clone(); | ||
| let inspection = tauri::async_runtime::spawn_blocking(move || { | ||
| crate::parquet_arrow::inspect(Path::new(&probe), None) |
There was a problem hiding this comment.
When the user selects explode, open_editable_rows can materialize one output row per list element, but this guard reuses inspect with the default preserve-as-JSON projection, so needsDecision is based only on the original row count and data bytes. A small file with a very large list column can therefore skip the confirmation and allocate far more rows than estimated; the guard should account for the selected options or force confirmation when any explode policy is present.
Useful? React with 👍 / 👎.
| if (isColumnarPath(path)) { | ||
| await get().openColumnarInspect(path); | ||
| return; |
There was a problem hiding this comment.
Project open applies each saved source by calling openPath, but this new route only opens the interactive Parquet/Arrow inspect dialog and returns without creating a tab. As a result, a project saved with a read-only columnar source restores with that tab missing until the user manually confirms the dialog, while the project baseline is still updated; project restore needs a non-interactive columnar open path or should bypass this dialog.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in e70bfab.
applyProjectPlan restored every source through openPath, which routes any Parquet/Arrow path into the interactive inspect dialog (openColumnarInspect) and returns without creating a tab — so a restored columnar source was left missing until the user manually confirmed the dialog, while the project baseline had already advanced.
Fix: project restore now routes columnar entries through a new non-interactive openColumnarRestore store action that reopens the source directly as an indexed read-only document with default policies (no dialog), awaiting the open job so the tab exists before the restore's tab-order / annotation / view-reapply steps run. Indexed read-only is the memory-safe, prompt-free reproduction of a columnar source: it never hits the F10 memory decision, and per the spec invariant an indexed document stays read-only unless explicitly converted.
The routing decision is extracted as a pure restoreOpenRoute() helper in src/lib/project.ts, covered by unit tests in project.test.ts (columnar extensions → columnarIndexed; csv/json/jsonl/tsv/zip/gz → standard).
Note: the project schema records no columnar open mode/policy (only delimiter/encoding/hasHeaderRow), so a faithful editable-vs-indexed round-trip would be a separate schema change; indexed read-only is the correct non-interactive default for restore. Gates green (npm lint/format/typecheck/build + 394 vitest; cargo fmt/clippy + 728 lib tests).
…ely on project open A project referencing a columnar (Parquet/Arrow) source restored it through openPath, which routes columnar files into the interactive inspect dialog and returns without creating a tab. The source was left missing until the user manually confirmed the dialog, even though the project baseline had already advanced — project restores must be non-interactive (F37). applyProjectPlan now routes columnar entries through a new openColumnarRestore action that reopens the source directly as an indexed read-only document with default policies (no dialog), awaiting the open job so the tab exists before the restore's tab-order / annotation / view-reapply steps run. Indexed read-only is the memory-safe, prompt-free reproduction of a columnar source and never hits the F10 memory decision (indexed documents stay read-only unless converted). The routing decision is extracted as a pure restoreOpenRoute() helper with unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # src/lib/project.test.ts # src/store/useStore.ts
1 participant
Summary
Adds first-class Apache Parquet and Apache Arrow interoperability — open and export typed columnar datasets while preserving types and nulls, with bounded memory on multi-gigabyte files.
Open (palette → "Open Parquet/Arrow…", drag-and-drop, or "Open file…" on
.parquet/.arrow/.feather/.ipc):u64::MAXround-trips losslessly), exact decimal precision+scale, floats, booleans, dates, timestamps with time-zone metadata, and UTF-8 all survive; a NULL stays distinct from an empty string end to end (editable opens preserve the distinction via collision-free per-column null tokens).Export (palette → "Export as Parquet/Arrow…"):
New backend modules
parquet_arrow.rs(+3282) andcolumnar_export.rs(+1442); frontend inspect + export dialogs, typed invoke wrappers, palette entries, and store wiring.Tests
cargo test --lib: 728 passed, 0 failed (up from 726 on the base — the rebase pulled in two newhighlight.rsunit tests from the parent's f42 Codex-fix). Includes 16parquet_arrowunit tests (type-matrix extremes round-trip, arrow file/stream parity, row-group-stat pruning vs full scan, struct flatten, list/map policies, timezone preservation, convert memory-check, cancellation cleanup, null-vs-empty) and 21columnar_exportunit tests (scoped exact rows/cols,u64::MAX→ uint64, decimal scale unification, compression codecs, subsecond → ns, zoned timestamps, unrepresentable → NULL + warning count, stale-revision rejection, null-vs-empty round-trip).vitest: 392 passed (42 files), including 19 incolumnar.test.ts.cargo fmt --check; clippy--all-targets --all-features -- -D warnings.Acceptance mapping
parquet_arrow.rs,ParquetInspectDialog.tsxarrow_ipc_file_and_stream_read_identicallyParquetInspectDialog.tsx,parquet_arrow.rsinspect_reports_shape_types_and_compression,convert_memory_check_uses_the_columnar_estimatedocument.rs,parquet_arrow.rscolumnar_documents_behave_like_indexed_read_only_documentsparquet_arrow.rsconvert_to_editable_assigns_collision_free_null_tokensu64::MAXlossless), decimal, float, bool, date, tz timestamps, UTF-8parquet_arrow.rs,columnar_export.rstype_matrix_round_trips_extremes,u64_max_selects_uint64_and_round_trips_losslessly,timestamp_time_zone_metadata_is_preserved,zoned_timestamp_preserves_zone_metadata_and_instantsparquet_arrow.rs,columnar_export.rsnull_vs_empty_string_round_trip_distinctly,columnar_document_re_export_preserves_null_vs_emptyparquet_arrow.rsstruct_flattening_is_deterministic_and_escapes_dots,list_and_map_policies_are_deterministic,explode_multiplies_rows_in_the_editable_openparquet_arrow.rs,filter.rsrow_group_stats_skip_groups_and_match_the_full_scan,row_group_size_applies_and_pruned_filtered_reads_match_full_scancolumnar_export.rs,ColumnarExportDialog.tsxscoped_export_writes_exact_rows_and_columns_in_order,parquet_compression_codecs_apply,arrow_file_and_stream_round_trip_like_parquet,cancelled_export_removes_all_outputcolumnar_export.rsunparseable_cells_export_null_with_warning_count,beyond_i64_with_negatives_warns_and_nulls,no_schema_exports_all_utf8,typed_false_keeps_cell_text_verbatimcolumnar_export.rs,parquet_arrow.rsstale_revision_is_rejected_before_writing,cancel_stops_open_inspect_and_convert_without_leftoversSTACKED PR
This PR is stacked on
feat/f42-highlight(its base branch), the tip of the chainmain ← f31 ← infra ← f33 ← f37 ← f38 ← f48 ← f40 ← f41 ← f42 ← … ← feat/f42-highlight.Review only the delta versus
feat/f42-highlight— the 5 commitsd8ebd5b…c9f53f8(parquet/arrow read side; export + commands; UI; gates + docs; and the integration marker). GitHub's "Files changed" against this base already scopes to that delta; the parent chain's commits are not part of this review.Integration: rebased cleanly onto
feat/f42-highlight(80dfd60) with no conflicts — the parent's new Codex-fix commit touched onlyhighlight.rs/Grid.tsx/HighlightRulesDialog.tsx, disjoint from every file f32 changes. No source fixes were required, sochore(f32): integrate onto chainis an empty marker commit.🤖 Generated with Claude Code