Bump iceberg-cpp to upstream main - #25
Draft
smaheshwar-pltr wants to merge 7 commits into
Draft
smaheshwar-pltr wants to merge 7 commits into
smaheshwar-pltr wants to merge 7 commits into
Conversation
Pin a new fork branch, duckdb-proto-iceberg: upstream main plus the two open upstream pull requests the extension needs (FileIO properties and the bundled S3 link dependencies) and a missing include that breaks MinGW builds. Upstream now covers the fork's other changes, credential vending and the public SnapshotUtil header. Adapt to upstream's API: RestCatalog is used through its Catalog view, and the S3 region property is client.region. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
REST catalogs can vend credentials as storage credentials scoped to location prefixes rather than in the table config. The old fork merged the best match into the FileIO properties; upstream keeps them separate. Overlay the credential with the longest prefix matching the table location, which resolves the TODO for the storage credentials field. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The old fork sent X-Iceberg-Access-Delegation: vended-credentials with every request; upstream does not, and some catalogs only vend credentials when asked. Upstream also reports scan metrics to the catalog by default, which would add a request per scan outside the catalog lock. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Upstream reads a boolean as true only for a case-insensitive "true", lets s3.ssl.enabled override the endpoint scheme, and matches storage credentials by canonical S3 prefix against full paths. Do the same, and match against the slash-terminated table location, so that DuckDB reads data files with the same settings and credentials iceberg-cpp uses for metadata. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With the Arrow 25 and AWS SDK versions iceberg-cpp now bundles, finalizing Arrow's S3 support at exit sometimes crashed: an AWS CRT event loop still shutting down on another thread logged through the SDK's logger after the SDK had torn it down. With logging off the SDK installs no logger. The extension defaults ARROW_S3_LOG_LEVEL to off before S3 is initialized, keeping a value the user set. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the iceberg-cpp pin to a new fork branch,
duckdb-proto-iceberg: upstreammainplus apache/iceberg-cpp#963 and apache/iceberg-cpp#964, the two open upstream PRs we still need, and the<cstdint>fix from smaheshwar-pltr/iceberg-cpp#24. Part of #1.RestCatalogis session-aware, so the extension uses it throughAsCatalog(), and the S3 region property is nowclient.region.X-Iceberg-Access-Delegation: vended-credentialsitself; the extension now sets it via aheader.property, since some catalogs only vend when asked. Upstream's new scan metrics reports to the catalog are turned off.ARROW_S3_LOG_LEVELtooffunless the user set it, so the SDK installs no logger.🤖 Generated with Claude Code