Rebuild Parquet.jl in pure Julia - #199
Conversation
Replace the legacy PAR2 runtime with bounded pure Julia metadata, schema, encoding, page, logical, nested reader, and writer layers. Add N5 and N6 conformance evidence, resource and mutation tests, documentation, CI, and the continuation handoff. Keep the package at preproduction status. Publication and oracle locking remain disabled until the remaining provenance and platform gates pass. BREAKING CHANGE: remove the legacy PAR2, cursor, reader, writer, and dataset implementation in favor of the new namespaced File, Table, and write API.
Adversarial review of the rewrite surfaced defects in metadata handling,
temporal round trips, and resource accounting. Fix them, then delete the
read and write paths that review proved unreachable.
Untrusted metadata
- Store preserved Thrift fields as `Vector{RawField}` instead of a tuple.
The tuple length was attacker controlled, and the generated `==`,
`isequal`, and `hash` specialize per length, so a small footer carrying
a few thousand unknown fields cost seconds of compilation on the first
comparison of decoded metadata. Re-check the union invariant when
encoding, because the vector is mutable after construction.
- Bound `max_schema_name_bytes` per operation instead of over the process
lifetime. One hostile file used to exhaust the shared budget and deny
every later file that carried a new column name.
- Reject a nested plan deeper than the reader's recursion limit with a
`LimitError`. The schema parser and plan compiler are iterative, so a
raised `max_metadata_depth` could otherwise exhaust the stack during
assembly and corrupt process state.
- Charge byte-stream-split fixed-width copies to the live budget, and
make the close guards atomic so a concurrent close cannot release a
materialization charge twice.
Temporal and rewrite fidelity
- Read a millisecond TIMESTAMP as `Timestamp{:millis}` when it is
adjusted to UTC. It previously decoded to a bare `DateTime`, so a read
and write cycle silently republished a UTC instant as local time.
Millisecond columns without adjustment still read as `DateTime`, and
the constructor now rejects the contradictory combination.
- Preserve `key_value_metadata` on a schema-bearing rewrite, with the
key and value sizes checked before allocation. Rewrites previously
dropped producer metadata such as `ARROW:schema`.
- Compare preserved fields exactly during provenance validation, so a
change to the raw Compact Thrift header state cannot pass as identical.
- Report an INT96 column as an unsupported feature rather than an invalid
file. INT96 is valid but deprecated, and is not implemented here.
Unreachable code
- Delete the flat LIST assembler in the table reader. Every schema
containing a group compiles to a nested plan, so the branch was
unreachable from `Parquet.Table`.
- Delete the MAP iterate fallbacks in the nested writer. A MAP value is
always a concrete `MapValue` or `AbstractDict`, both intercepted
earlier.
- Delete the unused column page builders and the unbudgeted dictionary
helpers they were the last callers of.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document the behavior that review found surprising but correct, and stop resolved environments from reaching the branch. - Note which annotations a read and write cycle does not preserve, and point at `LogicalColumn` for selecting an exact annotation. - State the reader's nesting depth bound and why the writer accepts deeper synthetic values than the reader will assemble. - Describe memory-mapped file lifetime: `close` releases the descriptor while the mapping waits for finalization, which can hold a Windows file open, and truncation by another process is fatal to the mapping. - Say what `max_materialized_bytes` covers, since borrowed vectors and mapped paths are not charged and an external source owns its storage. - Ignore the root and docs manifests and the corpus checkout. The conformance manifest under test/conformance/n6/julia stays tracked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The N6 evidence binds each producer to an exact source revision, so the preceding source changes invalidated it. Re-establish that binding by re-running the producers and the exact external gate, rather than by restating hashes. - Recompute the Parquet.jl producer descriptor over the new source using the gate's own composite algorithm. - Rebind the parquet-jl authority in the capability matrix. The producer harness refuses to emit evidence until the declared revision and toolchain match the descriptor. - Regenerate the producer evidence. The twelve generated Parquet files and ninety-five of the ninety-six evidence records are unchanged; only the authority record moved, which confirms the source changes leave the observable behavior identical. - Restate the capability digest each normalized evidence run record carries, and resolve the upstream evidence citations that follow from it. - Re-pin the frozen evidence digests and the artifact manifest. Verified with the exact external gate on macOS 15 arm64 against the pinned toolchain: 5117 of 5117 preproduction checks and 578 of 578 independent model checks, with the raw Java scanner, Arrow Rust, Parquet Java, PyArrow, and DuckDB oracles and the isolated schema validator. Gate status stays preproduction; publication and oracle locking remain unauthorized. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First continuous integration run of this branch exposed portability defects that only appear away from the development platform. - Install dependencies in the bounds job. It ran the N5 conformance suite without a build step, so the package failed to load. - Keep text files LF in the working tree. Windows checked out CRLF, which broke a multiline pattern in the N5 model tests. Conformance evidence and artifact manifests are content hashed, so a CRLF checkout would also change their digests. Binary fixtures are excluded, and no tracked file is renormalized by this change. - Compare a manifest path in its own forward-slash form. normpath emits the platform separator, so Windows rejected a correctly normalized path. - Loosen the writer scratch-stack allocation bounds. The same measurement runs about a seventh higher on x86-64 than on arm64, and the previous bound was tuned on arm64. Measured against the branch point on one machine the allocations are unchanged, so this is platform headroom rather than a regression. The bounds still catch a pass that stops reusing its scratch stacks, which costs far more than a constant per row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #199 +/- ##
==========================================
+ Coverage 86.46% 87.67% +1.20%
==========================================
Files 10 37 +27
Lines 1596 17875 +16279
==========================================
+ Hits 1380 15672 +14292
- Misses 216 2203 +1987 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
`close!` releases the descriptor while the mapping waits for finalization, so Windows keeps the file locked and removing the temporary file is refused. That is the documented source lifetime, so let this cleanup step accept the refusal instead of failing the suite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`relpath` emits the platform separator, so the harness looked for a backslash path in a manifest that records forward slashes and reported the included N6 model source as unpinned on Windows. The harness is itself pinned, so re-freeze the descriptor, the capability authority that names it, the producer evidence, the evidence bindings that cite it, and the artifact manifest. The source composite is unchanged, because the harness is a support file rather than package source. Re-verified with the exact external gate on macOS 15 arm64: 5117 of 5117 preproduction checks and 578 of 578 independent model checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The harness fsyncs its destination directory and relies on POSIX rename and symlink behavior, and refuses to publish anywhere else. The guard tests called it unconditionally, so Windows reported a refusal it was always going to make. Only the artifact manifest and its digest move; the producer descriptor, capability authority, and evidence are untouched, because this file is pinned as a static artifact rather than as producer source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This is very cool, but you should make a new package for this, because this PR is too big for people to review. |
Haha, I'm not sure a Parquet4.jl would go over well! No, I need to dig in to the diff here, because it's actually a way smaller set of actual code changes (comparable to the ARrow.jl package) and not actually 100K lines of julia to support parquet. I've only reviewed about half of it myself so far, so there's still probably ways to clean it up further. My understanding is that Parquet/Parquet2/Parquet3 are have various warts/gaps in spec coverage and none are super actively maintained. I'd like to have a really good package that does cover teh spec and can be pointed at as the "canonical" package; Parquet.jl seems like the best home for that. This isn't quite ready yet, but I'd love ot help anyone interested in helping digest a large codebase inot manageable chunks. |
|
Parquet2.jl is actively maintained and, right now I would call QuackIO.jl the canonical package. Parquet is hard because if you take this format seriously you should engage with upstream to ensure changes that are made to the format are compatible with the julia package. For example it would be good to see the hypothetical canonical julia package on https://parquet.apache.org/docs/file-format/implementationstatus/ to see how it stacks up against the standard implementations. Also as you can see from there, that nobody wants to have full spec coverage (there are deprecated things like LZO and one of the LZ4 codecs that you probably don't want to deal with). The spec is also vague and evolving over time, so in reality the target should be compatibility with Parquet2.jl, QuackIO.jl and the implementations in other languages. |
|
Also, is it possible to contribute improvements to https://github.com/JuliaAPlavin/QuackIO.jl/pulls and https://gitlab.com/ExpandingMan/Parquet2.jl/-/merge_requests instead of making another package? Maybe @aplavin @ExpandingMan or @bjarthur have some thoughts on this. |
|
I'm generally using QuackIO (admit I can be biased :) ), and would say its approach has both pros and cons. Obvious |
Is it? Last I heard, ExpandingMan was specifically not going to do further work on the package and was effectively "done" with it. I guess there are still dependabot bumps going on, but in my mind, it's not an actively developed package much more than Parqet.jl is. I have no issues/qualms with QuackIO.jl and I think it's an excellent option/format for certain use-cases. But it's also not what I'm personally interested in, which is pure-Julia parquet support. I'm also aware of the deprecated parts of the spec and don't plan to support them. Not trying to rock anyone's boat here or anything; I happen to have access to a lot of AI tokens at the moment, I have a lot of experience w/ data format implementations, and I've wanted a solid pure-Julia parquet for a long time with good mmap support, all hte Tables.jl integrations (including a new Tables.Scan component), etc. |
A MAP key snapshot exists only while a trace records one, so an aggregate scan run without a trace reached the key assertion with nothing to assert against and raised a MethodError. Struct and list sources already scanned without a trace; only MAP could not. Add the missing method for the absent snapshot. It keeps every check that does not need one, so only the comparison against a previous pass is skipped, and that comparison has no referent without a trace. The public writer always supplies a trace and never reached this. The regression covers both MAP scan arms, a dictionary source and a map view, and asserts that the scan still infers decimal and timestamp leaf parameters rather than merely completing. Re-freezing the N6 evidence follows from touching pinned package source. The twelve generated fixtures are unchanged. Re-verified with the exact external gate on macOS 15 arm64: 5117 of 5117 preproduction checks and 578 of 578 independent model checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The metadata types come from `julia thrift/generate.jl` and the N6 evidence records from the producer harness. Neither is edited by hand, so collapse both by default in review rather than reading them as authored code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Yes the Parquet2.jl package is very much alive. ExpandingMan is not driving development of new features, but last I checked he is still interested in supporting more Parquet features and willing to merge things and help maintain the code, your new way of handling thrift and all the java tests here seem like things that would be great in Parquet2.jl as well. From my perspective the only reason to keep using Parquet.jl is for people who like its API and don't want to learn Parquet2.jl or QuackIO.jl's API, so I can't see the reason to make breaking changes to the API of Parquet.jl IMO mmap often causes more problems then it is worth, and it is possible to match performance and have better error handling with manual caching systems (but I guess this depends on what operating system/file system/CPU you are using). |
|
Parquet2.jl is maintained in the sense that I intend to merge PR's and fix critical bugs, but yes it is very unlikely I will ever do more work than that on the package. I do think duckdb is a really good solution for parquet in Julia and it supports some fantastic features such as querying. In any case, I don't personally have any objections to Parquet2.jl being replaced with something incompatible. |
Complete coverage of the format was never a realistic goal, and treating it as one held the release behind work nobody has asked for. Decide these two out. LZO is stable and nondeprecated in the IDL, but every available implementation is GPL-2, so an MIT core cannot carry one. It was blocking the format-completeness claim behind a license-compatible implementation that does not exist. INT96 is deprecated in the format and was never implemented, although the roadmap claimed both directions. Neither makes a file invalid, so both now report an unsupported feature rather than an invalid file, matching how every other recognized-but-unsupported feature is reported. Record both as decided rather than pending, and state plainly that complete coverage is not a goal. The deprecated LZ4 codec and the deprecated BIT_PACKED encoding stay readable and stay unwritable. Files in the wild use them, so dropping the read path would cost compatibility rather than buy simplicity. Re-freezing the N6 evidence follows from touching pinned package source. Verified with the exact external gate on macOS 15 arm64: 5117 of 5117 preproduction checks and 578 of 578 independent model checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
4 participants
Rebuilds Parquet.jl as a pure Julia implementation, then applies the fixes from an
adversarial review of that rewrite.
The registered package UUID and the Git history are preserved. The old PAR2 runtime is
replaced. Read
HANDOFF.mdfor the branch contract before reviewing.What the rewrite provides
Bounded source ownership and footer framing; a pure Julia Thrift Compact Protocol
runtime with generated Parquet 2.13.0 metadata types; physical and logical schema
validation; Dremel levels and nested vectors; the PLAIN, RLE, dictionary, delta, and
byte-stream-split encodings; Data Page V1 and V2 framing with CRC checks; the
uncompressed, Snappy, gzip, Brotli, Zstd, and LZ4 codecs; scalar logical types with
bounded JSON, BSON, and decimal validation; recursive nested reading and writing;
statistics, producer-trust, and page-index handling; a Tables.jl facade and a
namespaced writer API.
The package has no exports. The public names are
BSONValue,Decimal,File,Interval,JSONValue,Limits,LogicalColumn,Table,Timestamp,close!,and
write.Review fixes in this branch
The library parses untrusted files, so the review treated every file-derived value as
attacker controlled.
Untrusted metadata. Preserved Thrift fields were stored in a tuple whose length was
attacker controlled; because the generated
==,isequal, andhashspecialize perlength, a small footer carrying a few thousand unknown fields cost seconds of
compilation on the first comparison of decoded metadata. They are now stored in a
Vector, and the union invariant is re-checked at encode time since that vector ismutable.
max_schema_name_bytesnow bounds a single operation instead of the processlifetime, so one hostile file can no longer exhaust a shared budget and deny every
later file carrying a new column name. A nested plan deeper than the reader's recursion
limit is rejected with a
LimitErrorrather than exhausting the stack, since theschema parser and plan compiler are iterative and could build such a plan under a
raised
max_metadata_depth. Byte-stream-split fixed-width copies are charged to thelive budget, and the close guards are atomic.
Temporal and rewrite fidelity. A millisecond
TIMESTAMPadjusted to UTC decoded toa bare
DateTime, so a read and write cycle silently republished a UTC instant aslocal time; it now reads as
Timestamp{:millis}. Legacy converted annotations arewritten for both values of
isAdjustedToUTC, matching the pinned IDL. Schema-bearingrewrites preserve
key_value_metadata, which previously dropped producer metadata suchas
ARROW:schema. Provenance validation compares preserved fields exactly, so a changeto raw Compact Thrift header state cannot pass as identical.
Unreachable code. The flat LIST assembler, the MAP iterate fallbacks in the nested
writer, and the unused column page builders were each proven unreachable from the
public API before removal. Coverage that only reached them through a dead wrapper was
retargeted onto the live entry points rather than deleted.
Every fix carries a focused regression test.
Verification
apache/parquet-testingcorpus present.5117 of 5117 preproduction checks and 578 of 578 independent model checks, exercising
the raw Java scanner, Arrow Rust, Parquet Java, PyArrow, and DuckDB oracles plus the
isolated schema validator. The evidence was re-established by re-running the
producers, not by restating hashes: the twelve generated Parquet files and
ninety-five of the ninety-six evidence records are byte-identical to before, which
independently confirms these changes leave observable behavior unchanged.
Status
This branch stays preproduction.
test/conformance/n6/manifest.tomlkeepsstatus = "preproduction", and bothpublication_authorizedandoracle_lock_authorizedremainfalse. Known remaining work is listed inHANDOFF.md, including LZO, the target-only modules, PyArrow and DuckDBsource-to-wheel provenance, and expanding the exact gate beyond macOS 15 arm64.
Note for CI: the N6 static lane verifies the running Julia runtime tree against a
frozen pin, which cannot be reproduced on an arbitrary local install. It is expected to
pass on a clean runner that downloads the pinned Julia versions.