Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,40 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

### Added

- **Parquet & Arrow interoperability** (palette → "Open Parquet/Arrow…" and
"Export as Parquet/Arrow…"): open and export typed columnar datasets —
Apache Parquet, Arrow IPC files (Feather v2 is the Arrow IPC file format),
and Arrow IPC streams — preserving types and nulls. Opening a `.parquet` /
`.arrow` / `.feather` / `.ipc` file (drag-and-drop, "Open file…", or the
command) first shows an inspect dialog: container format, row and
row-group/batch counts, compression codec, the schema mapped to the F31
logical types (nested fields indented, timezones shown), and an
editable-memory estimate — before anything is loaded. Choose read-only
(an indexed columnar backing with windowed reads over row groups / record
batches and a bounded decoded-block cache, so the grid, filters, and
export work with bounded memory on multi-gigabyte files) or convert to
editable behind an explicit memory check. Signed and unsigned 64-bit
integers (so `u64::MAX` round-trips losslessly), exact decimal
precision+scale, floats, booleans, dates, timestamps with their time-zone
metadata, and UTF-8 strings all survive intact, and a NULL stays distinct
from an empty string end to end (editable opens preserve the distinction
through collision-free per-column null tokens). Structs flatten to stable
path-based column names; each list/map/struct field takes an explicit
per-field policy (keep as JSON, explode into rows on an editable open, or
drop). Equality and range filters on numeric/date columns of indexed
parquet documents skip whole row groups using their statistics, with
results identical to a full scan. Export any scope (all rows, the filtered
view, selected rows / columns / range) to Parquet (uncompressed, Snappy,
or Zstd, with a configurable row-group size), an Arrow IPC file, or an
Arrow IPC stream, as a cancellable job through the atomic-save pipeline;
typed export maps each column's declared logical type to the matching
arrow type (Int64/UInt64, Decimal128 with unified precision/scale, Date32,
microsecond or nanosecond timestamps carrying the schema's time zone),
while null tokens and columnar NULLs export as real nulls distinct from
empty strings. Cells that cannot be represented under the declared types
are written as NULL and counted into a per-column warning report; columns
without a schema export as text verbatim. A columnar document opens
unsaved so a later Save can never overwrite the binary source with CSV.
- **Row bookmarks, tags & notes** (F40): mark and annotate records without
touching the source data. Star or flag a row, apply multiple named tags
(a per-document tag namespace with usage counts), and attach a row note or
Expand Down Expand Up @@ -199,6 +233,16 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
the backend), advisory accepts it and records a bounded, retrievable
issue — while schema edits themselves never touch the undo stack.

### Fixed

- **Project open now restores Parquet / Arrow sources non-interactively**:
reopening a project that referenced a columnar source no longer routes it
through the interactive inspect dialog (which returned without creating a
tab, so the source was left missing until the user manually confirmed the
dialog while the project baseline had already advanced). A restore reopens a
columnar source directly as an indexed read-only document with default
policies, matching the non-interactive restore of every other source type.

### Internal

- **Shared tabular contracts**: new backend `TabularSource` /
Expand Down
15 changes: 15 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,15 @@ and faithful on large, real-world delimited files.**
detects the shape, infers columns, counts missing vs explicit-null cells,
and lets you flatten, preserve, join, or explode nested objects and
arrays. JSON Lines opens read-only with bounded memory.
- **Open Parquet / Arrow** — inspect an Apache Parquet, Arrow IPC file
(Feather v2), or Arrow IPC stream before opening: format, row and
row-group/batch counts, compression codec, the schema mapped to logical
types (with nested fields and timezones), and the editable-memory
estimate. Open read-only (indexed, bounded memory) or convert to editable;
nested list/map/struct fields take a per-field policy (keep as JSON,
explode into rows, or drop). Signed/unsigned 64-bit integers, exact
decimal precision/scale, timestamps with timezone, and null-vs-empty-string
all survive intact.
- Auto-detect the **delimiter** (comma, tab, semicolon, pipe) with a manual /
custom override — plus an **advanced import** for preambles, comment
lines, custom quoting/escaping, multi-row headers, and footers.
Expand Down Expand Up @@ -106,6 +115,12 @@ and faithful on large, real-world delimited files.**
array of arrays, or JSON Lines; typed columns emit real numbers and
booleans, nested objects rebuild from dotted-path columns, and duplicate
output paths are rejected before writing.
- **Export as Parquet / Arrow** — write any export scope to Apache Parquet
(uncompressed, Snappy, or Zstd, with a configurable row-group size), an
Arrow IPC file (Feather v2), or an Arrow IPC stream. Typed export maps each
column's declared logical type to the matching arrow type (preserving 64-bit
integer widths, decimal precision/scale, and timestamp timezones); cells
that can't be represented are written as NULL and reported per column.

**Reliability**

Expand Down
Loading
Loading