Repository navigation
[FEATURE][Java] Adapt Parquet physical IO to columnar batches #964
Description
Describe the enhancement requested
After #959 / PR #961 replaces the provisional row-oriented physical-batch contract with
column vectors, the Parquet backend needs the corresponding adapter migration. This issue is one
narrow downstream slice: it must not retain a compatibility Row facade.
Scope:
- migrate
graphar-io-apiread-contract tests andgraphar-io-parquetfrom
RecordBatch.row(int)/RowtoValueVectorandVectorRecordBatch; - preserve physical projection, strict half-open row-range reads, and required Parquet Offset Index
behavior; - preserve safe binary/list/null value ownership at the vector boundary;
- treat
WriteMode.APPENDexplicitly: Parquet cannot atomically append to one existing file, so the
writer must reject it rather than silently overwriting.
Non-goals: migrating GraphAr reader/writer/validator/integration consumers (separate dependent
slices); adding Apache Arrow as a dependency; changing the Parquet decoding algorithm; or making
Parquet append work through a hidden rewrite.
Acceptance evidence:
- no
Rowor.row(...)reference remains inio-api/io-parquet; - focused clean verification covers vector schema position, count, null, binary and list values;
- canonical fixture projection/row-range tests retain applied
PROJECTIONandROW_RANGE; - writer still emits readable Offset Indexes with the existing bounded-page assertion;
- a write with
APPENDis explicitly rejected, never treated as overwrite.
Component(s): Java
Activity
keksmd commentedon Sep 8, 2026
Status and the exact gate on this slice.
The module is ready in migrated form — no `Row` and no `.row(...)` anywhere in it; batches are `ValueVector` / `VectorRecordBatch`, and `ParquetRow` is replaced by `ParquetValueVector`. I verified it compiles and tests clean against
mainplus the read contract from #972:It cannot be opened as a PR against
maintoday, and the reason is a real dependency, not polish: the reader side usesPhysicalReader,ReadRequest,ReadResult,ReadReport,Projection,RowRange,Filter,LiteralandComparisonOperator, and none of those exist onmainyet — they are #971 and #972. Basing this PR on an unmerged branch would make a dependent chain, so I would rather wait: once #971 and #972 land, this goes up as a standalone PR cut frommainthe same day.Ordering for the remaining Java SDK slices, so nobody has to reconstruct it from the PR queue:
@SemyonSinchenko @SYaoJun review bandwidth on #971 and #972 would unblock the format backend, which is the layer that makes the storage work merged so far actually readable end to end.
keksmd commentedon Sep 9, 2026
Status after #968 landed: this is now the single gate for everything else in the Java SDK — the reader, the writer (#973) and the validator (#974) all verify their output by reading it back through a Parquet backend.
Its remaining dependencies are #971 and #972 (
ReadRequest,PhysicalReader,ReadReport). Both are green and rebased on currentmain, waiting on review. I am holding this PR until they merge rather than opening a third link in a chain; the branch is ready and builds againstmainplus those two.