fix(optimizer): Resolve bucketing per scan, not per table (#1812) - #1812
Open
pradeepvaka wants to merge 1 commit into
Open
fix(optimizer): Resolve bucketing per scan, not per table (#1812)#1812pradeepvaka wants to merge 1 commit into
pradeepvaka wants to merge 1 commit into
Conversation
meta-cla
Bot
added
the
CLA Signed
meta-codesync
Bot
changed the title
pradeepvaka
added a commit
to pradeepvaka/verax
that referenced
this pull request
Sep 1, 2026
…cubator#1812) Summary: Tables can be rebucketed without rewriting older partitions. A filtered scan may therefore read partitions that do not match the table's current bucketing, even though planning treated the layout as a guarantee. After connector filters are pushed, the optimizer asks for the exact selected partitions and their storage bucketing. The scan keeps that result, and distribution planning uses the connector's guarantee instead of assuming the table definition describes every selected partition. Connectors that only participate in planning can continue to use their declared layout without registering execution metadata. Connectors that support partition selection reuse the same result for estimates and final scan emission, so planning decisions stay aligned. Differential Revision: D115056959
pradeepvaka
force-pushed
the
export-D115056959
branch
from
September 1, 2026 04:28
3cf3d71 to
8e4e404
Compare
pradeepvaka
added a commit
to pradeepvaka/verax
that referenced
this pull request
Sep 1, 2026
…cubator#1812) Summary: X-link: https://github.com/facebookincubator/axel/pull/294 Tables can be rebucketed without rewriting older partitions. A filtered scan may therefore read partitions that do not match the table's current bucketing, even though planning treated the layout as a guarantee. After connector filters are pushed, the optimizer asks for the exact selected partitions and their storage bucketing. The scan keeps that result, and distribution planning uses the connector's guarantee instead of assuming the table definition describes every selected partition. Connectors that only participate in planning can continue to use their declared layout without registering execution metadata. Connectors that support partition selection reuse the same result for estimates and final scan emission, so planning decisions stay aligned. Differential Revision: D115056959
pradeepvaka
force-pushed
the
export-D115056959
branch
from
September 1, 2026 04:45
8e4e404 to
79c68d2
Compare
pradeepvaka
added a commit
to pradeepvaka/verax
that referenced
this pull request
Sep 1, 2026
…cubator#1812) Summary: X-link: https://github.com/facebookincubator/axel/pull/294 Tables can be rebucketed without rewriting older partitions. A filtered scan may therefore read partitions that do not match the table's current bucketing, even though planning treated the layout as a guarantee. After connector filters are pushed, the optimizer asks for the exact selected partitions and their storage bucketing. The scan keeps that result, and distribution planning uses the connector's guarantee instead of assuming the table definition describes every selected partition. Connectors that only participate in planning can continue to use their declared layout without registering execution metadata. Connectors that support partition selection reuse the same result for estimates and final scan emission, so planning decisions stay aligned. Differential Revision: D115056959
pradeepvaka
force-pushed
the
export-D115056959
branch
from
September 1, 2026 05:23
79c68d2 to
b8b5606
Compare
pradeepvaka
added a commit
to pradeepvaka/verax
that referenced
this pull request
Sep 1, 2026
…cubator#1812) Summary: X-link: https://github.com/facebookincubator/axel/pull/294 Tables can be rebucketed without rewriting older partitions. A filtered scan may therefore read partitions that do not match the table's current bucketing, even though planning treated the layout as a guarantee. After connector filters are pushed, the optimizer asks for the exact selected partitions and their storage bucketing. The scan keeps that result, and distribution planning uses the connector's guarantee instead of assuming the table definition describes every selected partition. Connectors that only participate in planning can continue to use their declared layout without registering execution metadata. Connectors that support partition selection reuse the same result for estimates and final scan emission, so planning decisions stay aligned. Differential Revision: D115056959
pradeepvaka
force-pushed
the
export-D115056959
branch
2 times, most recently
from
September 1, 2026 06:21
9f95d6a to
2cc60d2
Compare
meta-codesync
Bot
changed the title
|
@pradeepvaka has exported this pull request. If you are a Meta employee, you can view the originating Diff in D115056959. |
meta-codesync
Bot
changed the title
pradeepvaka
added a commit
to pradeepvaka/verax
that referenced
this pull request
Sep 1, 2026
…cubator#1812) Summary: X-link: https://github.com/facebookincubator/axel/pull/294 Tables can be rebucketed without rewriting older partitions. A filtered scan may therefore read partitions that do not match the table's current bucketing, even though planning treated the layout as a guarantee. After connector filters are pushed, the optimizer asks for the exact selected partitions and their storage bucketing. The scan keeps that result, and distribution planning uses the connector's guarantee instead of assuming the table definition describes every selected partition. Connectors that only participate in planning can continue to use their declared layout without registering execution metadata. Connectors that support partition selection reuse the same result for estimates and final scan emission, so planning decisions stay aligned. Differential Revision: D115056959
pradeepvaka
force-pushed
the
export-D115056959
branch
from
September 1, 2026 07:12
2cc60d2 to
e4aa7ef
Compare
meta-codesync
Bot
changed the title
pradeepvaka
force-pushed
the
export-D115056959
branch
2 times, most recently
from
September 1, 2026 07:57
8caeacb to
d1c7549
Compare
meta-codesync
Bot
changed the title
pradeepvaka
added a commit
to pradeepvaka/verax
that referenced
this pull request
Sep 1, 2026
…cubator#1812) Summary: X-link: https://github.com/facebookincubator/axel/pull/294 Tables can be rebucketed without rewriting older partitions. A filtered scan may therefore read partitions that do not match the table's current bucketing, even though planning treated the layout as a guarantee. After connector filters are pushed, the optimizer asks for the exact selected partitions and their storage bucketing. The scan keeps that result, and distribution planning uses the connector's guarantee instead of assuming the table definition describes every selected partition. Connectors that only participate in planning can continue to use their declared layout without registering execution metadata. Connectors that support partition selection reuse the same result for estimates and final scan emission, so planning decisions stay aligned. Differential Revision: D115056959
pradeepvaka
added a commit
to pradeepvaka/verax
that referenced
this pull request
Sep 1, 2026
…cubator#1812) Summary: X-link: https://github.com/facebookincubator/axel/pull/294 Tables can be rebucketed without rewriting older partitions. A filtered scan may therefore read partitions that do not match the table's current bucketing, even though planning treated the layout as a guarantee. After connector filters are pushed, the optimizer asks for the exact selected partitions and their storage bucketing. The scan keeps that result, and distribution planning uses the connector's guarantee instead of assuming the table definition describes every selected partition. Connectors that only participate in planning can continue to use their declared layout without registering execution metadata. Connectors that support partition selection reuse the same result for estimates and final scan emission, so planning decisions stay aligned. Differential Revision: D115056959
pradeepvaka
force-pushed
the
export-D115056959
branch
2 times, most recently
from
September 1, 2026 14:49
25304f7 to
9ae0f5e
Compare
pradeepvaka
added a commit
to pradeepvaka/verax
that referenced
this pull request
Sep 1, 2026
…cubator#1812) Summary: X-link: https://github.com/facebookincubator/axel/pull/294 Tables can be rebucketed without rewriting older partitions. A filtered scan may therefore read partitions that do not match the table's current bucketing, even though planning treated the layout as a guarantee. After connector filters are pushed, the optimizer asks for the exact selected partitions and their storage bucketing. The scan keeps that result, and distribution planning uses the connector's guarantee instead of assuming the table definition describes every selected partition. Connectors that only participate in planning can continue to use their declared layout without registering execution metadata. Connectors that support partition selection reuse the same result for estimates and final scan emission, so planning decisions stay aligned. Differential Revision: D115056959
pradeepvaka
force-pushed
the
export-D115056959
branch
from
September 1, 2026 14:51
9ae0f5e to
044fc71
Compare
…cubator#1812) Summary: X-link: https://github.com/facebookincubator/axel/pull/294 Tables can be rebucketed without rewriting older partitions. A filtered scan may therefore read partitions that do not match the table’s current bucketing, even though planning treated the layout as a guarantee. After connector filters are pushed, the optimizer asks for the exact selected partitions and their storage bucketing. The scan keeps that result, and distribution planning uses the connector’s guarantee instead of assuming the table definition describes every selected partition. Connectors that only participate in planning can continue to use their declared layout without registering execution metadata. Connectors that support partition selection reuse the same result for estimates and final scan emission, so planning decisions stay aligned. Partition selection records planning wall time, CPU time, and selected-partition count separately from runtime fallback listing. Axel publishes both paths through the existing partition-resolution time-series counter. Differential Revision: D115056959
pradeepvaka
force-pushed
the
export-D115056959
branch
from
September 1, 2026 15:47
044fc71 to
58cc077
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
X-link: https://github.com/facebookincubator/axel/pull/294
Tables can be rebucketed without rewriting older partitions. A filtered scan may therefore read partitions that do not match the table’s current bucketing, even though planning treated the layout as a guarantee.
After connector filters are pushed, the optimizer asks for the exact selected partitions and their storage bucketing. The scan keeps that result, and distribution planning uses the connector’s guarantee instead of assuming the table definition describes every selected partition.
Connectors that only participate in planning can continue to use their declared layout without registering execution metadata. Connectors that support partition selection reuse the same result for estimates and final scan emission, so planning decisions stay aligned.
Partition selection records planning wall time, CPU time, and selected-partition count separately from runtime fallback listing. Axel publishes both paths through the existing partition-resolution time-series counter.
Differential Revision: D115056959