Skip to content

Modular Encryption V2 - #782

Closed
kiloOhm wants to merge 12 commits into
aloneguid:masterfrom
kiloOhm:modular-encryption-6-1
Closed

kiloOhm wants to merge 12 commits into
aloneguid:masterfrom
kiloOhm:modular-encryption-6-1

Conversation

@kiloOhm

@kiloOhm kiloOhm commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Rebuilt from current master as suggested in #654

Summary

Adds Parquet Modular Encryption support to parquet-dotnet across the reader and writer paths.

What changed

  • Added public encryption and decryption configuration through ParquetOptions.
  • Supports AES_GCM_V1 and AES_GCM_CTR_V1.
  • Supports encrypted footers and signed plaintext footers.
  • Supports uniform footer-key encryption and per-column keys.
  • Supports 128-, 192-, and 256-bit AES keys.
  • Supports stored and externally supplied AAD prefixes.
  • Added key resolution through direct keys or IParquetKeyRetriever.
  • Encrypts and decrypts footers, column metadata, page headers, and page bodies.
  • Allows reading available plaintext/projected columns without every column key.
  • Added decryption-aware ReadSchemaAsync overloads.
  • Fails closed on wrong keys, tampered modules, mismatched AAD prefixes, and unknown configured column paths.
  • Explicitly rejects appending to encrypted files, which is not supported yet.

Interoperability and testing

  • Added parquet-java/parquet-mr interoperability coverage.
  • Added external fixture coverage across:
    • both encryption algorithms
    • encrypted and plaintext footer modes
    • stored, supplied, and absent AAD prefixes
    • uniform and partial column encryption
    • 128- and 256-bit keys
    • key metadata configurations
  • Replaced the committed 79 MB inspector JAR with pinned, reviewable Java source built by Maven in CI.
  • Added DuckDB interop test Encryption support (footer key) #726

Validation completed with:

dotnet test src/Parquet.slnx -c Release --no-restore
895 passed, 3 skipped, 0 failed

Lukas Wicke added 8 commits August 11, 2026 20:53
Accept authenticated zero padding emitted by parquet-mr while rejecting non-zero trailing data and invalid footer lengths.
Exercise both reader and writer compatibility with the parquet-mr fixture matrix and specialized Java inspector.
Reject mismatched stored AAD prefixes and unknown encryption column paths so file identity and confidentiality cannot silently degrade.

Expose decryption-aware schema reads and build the parquet-java interop inspector from pinned, reviewable source instead of committing the generated fat JAR.
@kiloOhm
kiloOhm marked this pull request as ready for review August 11, 2026 20:39
Lukas Wicke added 4 commits August 11, 2026 23:01
Column-key resolution may depend on the physical column path. Preserve metadata-only retrievers through a default interface overload while passing null for footer resolution.
@kiloOhm

kiloOhm commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Just read the LLM rule. Closing

@kiloOhm kiloOhm closed this Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant