-
Notifications
You must be signed in to change notification settings - Fork 4.3k
GH-48701: [C++][Parquet] Add ALPpd encoding #48345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
prtkgaur
wants to merge
122
commits into
apache:main
Choose a base branch
from
prtkgaur:gh540-alp-pseudoDecimal-encoding
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
122 commits
Select commit
Hold shift + click to select a range
ae3d27e
Add alp code
sfc-gh-pgaur 19968d2
Integrate ALP with arrow
sfc-gh-pgaur 2274077
Add alp benchmark
sfc-gh-pgaur 06a4027
Add datasets for alp benchmarking
sfc-gh-pgaur 3f25cc3
Update cmake file
sfc-gh-pgaur 3b0a0d7
Move hpp files to h
sfc-gh-pgaur 93f291c
Update flow digram and layout digram to use ASCII and not unicode cha…
sfc-gh-pgaur 6931ca5
Rename cpp files to cc
sfc-gh-pgaur 2a62002
Update documentation to align with arrow's doxygen style
sfc-gh-pgaur eaef4d0
Adapt methods and variable names to arrow style
sfc-gh-pgaur c62111e
Update the tests to adhere to arrow style code
sfc-gh-pgaur 2832385
Update callers
sfc-gh-pgaur eb2ce54
Fuse FOR and decode loop
sfc-gh-pgaur 66b59e0
Reduce memory allocation in the decompress call
sfc-gh-pgaur 7db0f18
Attempt at making decoding faster with SIMD
sfc-gh-pgaur eab7cae
Revert "Attempt at making decoding faster with SIMD"
sfc-gh-pgaur 57ec026
Move cpp files to cc
sfc-gh-pgaur af324cb
Move data file to parquet-testing submodule
sfc-gh-pgaur eadc951
Update path to the data file
sfc-gh-pgaur 80f9d98
Adapt files names to arrow convention
sfc-gh-pgaur 2629bda
File rename
sfc-gh-pgaur 595bf65
Obtain compressed size and number of elements from page header
sfc-gh-pgaur f2a7e90
Fix namespace depth
sfc-gh-pgaur da142b6
Better pack the compression block header
sfc-gh-pgaur 0f896e8
Rename class
sfc-gh-pgaur 3be05a2
Rearrage field for vector metadata for better packing
sfc-gh-pgaur f07c5cc
Add spec files
sfc-gh-pgaur 29a513b
Add more tests
sfc-gh-pgaur a68121f
Move number of elements metadata info to uber header
sfc-gh-pgaur c500689
Keep version as 1
sfc-gh-pgaur 77caa8c
Tests pass
sfc-gh-pgaur 42b4601
Get rid of numElements and derive it for each vector
sfc-gh-pgaur a4f02f2
Update the md files
sfc-gh-pgaur fa2e820
Fix ubsan error
sfc-gh-pgaur 0098a01
Use safe copy
sfc-gh-pgaur 9524032
Add more unit tests
sfc-gh-pgaur 8b1447b
Templatize AlpEncodedVectorInfo
sfc-gh-pgaur 15b0344
Fix
sfc-gh-pgaur 8e13cc5
[SpecReview] Num values in page is at max int32
sfc-gh-pgaur 596340b
[SpecReview] Rename layout to IntegerEncoding
sfc-gh-pgaur 63015f5
[SpecReview] Rename enum
sfc-gh-pgaur 154ee3a
[SpecReview] Make AlpHeader exact 8 bytes
sfc-gh-pgaur 318d79b
[SpecReview] Move all vector data towards the start
sfc-gh-pgaur a5c3623
[SpecReview][Metadata] Introduce wrapper to read metadata
sfc-gh-pgaur 1ad5d78
[SpecReview] Separate out ALP and FOR metadata
sfc-gh-pgaur 6b2fba0
[SpecReview] Remove unnecessary field
sfc-gh-pgaur 8629a9f
[FutureFlexibility] Add the FOR specific code behind switch
sfc-gh-pgaur c592e09
Add unit test
sfc-gh-pgaur 6f159a7
ALP: Implement offset-based layout for O(1) random access
sfc-gh-pgaur 69d7e5c
ALP: Update documentation for offset-based layout
sfc-gh-pgaur 95e020a
ALP: Add CreateSamplingPreset and EncodeWithPreset APIs
sfc-gh-pgaur 31d2701
ALP: Update all comments and docs for offset-based layout
sfc-gh-pgaur f845dac
ALP: Add cross-implementation reference blob generator
sfc-gh-pgaur 0e2fc95
ALP: Add generate_alp_parquet utility for creating ALP test parquet f…
sfc-gh-pgaur afdec0d
ALP: Remove version field from header (8B → 7B)
sfc-gh-pgaur a678c96
Update parquet-testing submodule with ALP test data files
sfc-gh-pgaur f36110f
ALP: Add C++ correctness tests for ALP-encoded parquet files
sfc-gh-pgaur 64607ca
ALP: Add cross-language tests reading Java-generated ALP parquet files
sfc-gh-pgaur e015b6d
ALP: Add float32 cross-language tests and generator support
sfc-gh-pgaur 57f4c2f
Address code review comments
sfc-gh-pgaur 9fffec8
Address code review comments: endianness assert and decode bounds checks
sfc-gh-pgaur 35c7fe4
Address code review comments: renames, API cleanup, and simplifications
sfc-gh-pgaur ad0d639
Fix build errors from code review changes
sfc-gh-pgaur 39dd599
Split EstimateCompressedSize loop into batched passes for vectorization
sfc-gh-pgaur 2f78bb0
Address review nits: uint64_t→int64_t, spell out ALP, use uint8_t for…
sfc-gh-pgaur 4586802
Move decoder creation outside benchmark loop
sfc-gh-pgaur 38aa1ec
Rename alp_wrapper.h/.cc to alp_codec.h/.cc
sfc-gh-pgaur 86ec59c
Convert ARROW_CHECK to Status returns on decode/load paths
sfc-gh-pgaur 6d9cb43
Add precondition docs to Store methods
sfc-gh-pgaur 92dee2c
Use signed integer types consistently per Arrow style guide
sfc-gh-pgaur 2469b22
Move out parameters to last position and remove unused enforce_mode
sfc-gh-pgaur 76c85fa
Add ALP encoding to parquet.thrift Encoding enum
sfc-gh-pgaur 7773c8e
Remove ALP spec files; spec belongs in parquet-format repo
sfc-gh-pgaur edda0c5
Rename decomp/comp params to input/output; remove ARROW_CHECK(false) …
sfc-gh-pgaur dc1e9bc
Copy header fields individually instead of bulk memcpy
sfc-gh-pgaur d3ea5a0
Replace StaticVector with std::vector for flexible vector size
sfc-gh-pgaur 1996ea5
Make ALP vector_size configurable on the encode path
sfc-gh-pgaur fab5d30
Add parquet-level round-trip test for ALP at non-default vector sizes
sfc-gh-pgaur 9cf3d3c
Address reviewer feedback: use int64_t, reorder out-params last, clea…
sfc-gh-pgaur f96e902
Add sign branching to FastRound for correct rounding of negative values
sfc-gh-pgaur 2c310d8
Address reviewer feedback: SafeLoad/SafeStore, struct→class conversion
sfc-gh-pgaur 275b93b
Convert AlpEncodedVector and AlpEncodedVectorView to private members …
sfc-gh-pgaur 4b28dc2
Remove unused AlpMetadataCache class and redundant GetNumElements()
sfc-gh-pgaur c805e9a
Remove private inheritance from AlpCompression and AlpInlines
sfc-gh-pgaur d62c475
Replace trailing return type with explicit Result<AlpHeader> in LoadH…
sfc-gh-pgaur df08fb2
Use SafeLoadAs for all header fields in LoadHeader
sfc-gh-pgaur 20f6394
Validate header fields in LoadHeader
sfc-gh-pgaur b62c063
Change AlpCodec public API from byte size to element count
sfc-gh-pgaur 5dc4032
Use uint8_t* consistently in AlpCodec API instead of char*
sfc-gh-pgaur ef1ea11
Replace trailing return types with explicit return types
sfc-gh-pgaur 620b7b3
Cite ALP paper §3.1.2 for combination tie-break rule
sfc-gh-pgaur 2a8aa77
Pin AlpMode underlying type to uint8_t for serialization safety
sfc-gh-pgaur 12f31e3
Address reviewer nits: doc tweaks, sizeof(member_), incremental-decod…
sfc-gh-pgaur 89460ea
Convert AlpSampler uint64_t to int64_t
sfc-gh-pgaur 3777241
Remove GH-48701 reference from incremental-decode TODO
sfc-gh-pgaur b96dea5
Document actual crash/abort behavior in AlpCodec encode-path docs
sfc-gh-pgaur 6c2a109
Use bit_util::CeilDiv for vector-count computations in AlpCodec
sfc-gh-pgaur 6e792d2
Replace memcmp with IsBitwiseEqual helper in alp_test.cc
sfc-gh-pgaur 81ffb57
Type-parameterize alp_test integration tests and fix TruncatedData
sfc-gh-pgaur 1869d9f
Type-parameterize AlpEncodingAdHoc tests in encoding_test.cc
sfc-gh-pgaur 9da0605
Widen RandomData test range to exercise ALP exception fallback path
sfc-gh-pgaur 81d9cba
Remove uncompiled generate_alp_parquet and generate_reference_blobs
sfc-gh-pgaur 60c3d89
Add incremental-encode TODO and snake_case rename in AlpEncoder
sfc-gh-pgaur e20e3d1
Rename EncodeAlp parameter combinations to preset
sfc-gh-pgaur defe57a
Use bit_util::IsPowerOf2 and BytesForBits instead of raw bit manipula…
sfc-gh-pgaur ba08de2
Remove remaining 'non-templated' references in ALP comments
sfc-gh-pgaur bb64c79
Remove GetBitPackedSize wrapper, inline bit_util::BytesForBits at cal…
sfc-gh-pgaur c3f5b00
Code review changes
sfc-gh-pgaur a99ae06
Convert AlpCodec public API from ARROW_CHECK aborts to Status returns
sfc-gh-pgaur 504dcea
Use arrow::csv::TableReader for ALP benchmark dataset loading
sfc-gh-pgaur 899d2e6
Reorder AlpEncoded(For)VectorInfo so kStoredSize uses sizeof(member_)
sfc-gh-pgaur 43eb5a7
Make ALP num_exceptions and exception positions unsigned
sfc-gh-pgaur 5c6706c
Enforce the spec's [3, 15] log_vector_size range in ALP
sfc-gh-pgaur 9bd4b6f
Use array matchers in ALP tests; note PLAIN fallback
sfc-gh-pgaur 7b7c1fd
Document the [3, 15] vector_size range in AlpCodec docs
sfc-gh-pgaur 78bde38
Put EncodeAlp output parameters last
sfc-gh-pgaur c9d6196
Test that ALP rejects malformed page headers
sfc-gh-pgaur fc53d86
Use std::countr_zero instead of __builtin_ctz in ALP
sfc-gh-pgaur bb414c9
Apply clang-format 18 to the ALP sources
sfc-gh-pgaur 8647b0d
Validate ALP vector metadata before it steers the decode
sfc-gh-pgaur 021e4f9
Correct inaccurate ALP comments; drop unused UnsafeResize
sfc-gh-pgaur fc74a1e
Type-parameterize ALP compression tests; tighten round-trip asserts
sfc-gh-pgaur File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the reviewer rough stats about this PR :
Production Code | ~2,449 | ~39%
Test Code. | ~1,151+ | ~18%
Benchmark | ~1,824 | ~29%
Documentation. | ~897 | ~14%