Skip to content

feat(inspect): capture scan inputs and open snapshot tasks - #1120

Draft
manuzhang wants to merge 5 commits into
apache:mainfrom
manuzhang:codex/metadata-scan-support-20261115
Draft

manuzhang wants to merge 5 commits into
apache:mainfrom
manuzhang:codex/metadata-scan-support-20261115

Conversation

@manuzhang

@manuzhang manuzhang commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Purpose

Regular data-table scans already support snapshot selection through UseSnapshot(), UseRef(), and AsOfTime(). MetadataScanBuilder also already exposes these selectors, but its implemented metadata tables (snapshots, history, and refs) use in-memory static tasks and reject time-travel selection.

This PR adds a generic snapshot-task execution path for the upcoming FilesTable (#876), ManifestsTable (#1121), and PartitionsTable (#1122) implementations. Those tables need to read metadata rows for a selected snapshot. It reuses the existing selector API and snapshot lookup utilities; the table implementations follow in their own PRs.

For those new tables, a source-table refresh after NewScan() must not change the metadata or schema used by that builder's scans. Built scans and tasks must retain their inputs after the metadata table or task reader is released. This PR supplies that ownership contract and builds directly on main.

Changes and rationale

  • Capture consistent inputs. Capture metadata and FileIO when creating the builder, then derive the schema through SchemaForScan(captured_metadata). This lets tables with evolving schemas use the same metadata version for their schema and rows.
  • Share snapshot-task execution. Add SnapshotMetadataTask to retain the scan context and a row-reader function pointer. MetadataScanTaskReader opens the task through that callback, so each metadata table can supply its reader without adding a table-specific dispatch branch.
  • Validate before reading. Check whole-column projections and resolve snapshot IDs, references, or timestamps against the captured metadata when building the scan. Reference lookup reuses SnapshotUtil::LatestSnapshot() after checking that a requested reference exists and is non-null; a main-branch selection with no current snapshot resolves to nullptr, while dangling snapshot IDs remain errors. Tasks can be reopened with independent streams and configurable batch sizes.

Public API and ABI changes

The public virtual getter changes its return type:

- virtual const std::shared_ptr<Schema>& schema() const = 0;
+ virtual std::shared_ptr<Schema> schema() const = 0;

The member function remains const. Returning the pointer by value lets implementations copy it under their schema-cache lock, giving callers ownership that remains valid when the cache is replaced. This copies the shared_ptr, not the Schema.

Derived classes must update their overrides, some reference-binding callers need changes, and existing binary consumers must be rebuilt. The PR also adds the protected virtual SchemaForScan() hook and TaskKind::kSnapshot.

Validation

  • Standalone PR: 59 metadata tests and C++20 compatibility passed; pre-commit and clang-tidy 22.1.8 passed.
  • Full split composition: 145 tests passed with one expected format-version skip; C++20 compatibility passed.
  • Additional shared-library check: the build and C++20 linkage passed. Shared Debug manifest IO tests hit an Avro std::bad_cast while writing fixtures, before the new helper runs; the same failure reproduces in unchanged existing manifest-writer tests. The full shared runtime suite is therefore not reported as passing.

Copilot AI balanced review requested due to automatic review settings November 14, 2026 16:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Extract a focused piece from the metadata table implementation with its tests.

Co-authored-by: Codex <codex@openai.com>
@manuzhang
manuzhang force-pushed the codex/metadata-scan-support-20261115 branch from f11adff to 33c84fb Compare November 15, 2026 15:26
Copilot AI balanced review requested due to automatic review settings November 15, 2026 15:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@manuzhang manuzhang changed the title feat(inspect): add shared snapshot metadata scan support feat(inspect): capture scan inputs and open snapshot tasks Nov 15, 2026
Remove the private SnapshotAtRef helper. Validate explicit references and
handle an empty main snapshot before delegating to LatestSnapshot, so
missing references and dangling snapshot IDs still fail.

Cover empty tables, missing/null references, and dangling snapshot IDs.

Co-authored-by: Codex <codex@openai.com>
@manuzhang
manuzhang marked this pull request as draft November 16, 2026 15:18
manuzhang and others added 3 commits November 17, 2026 19:32
Extract SnapshotUtil::IsMainBranch and reuse it for metadata snapshot selection, schema lookup, and latest snapshot lookup.

Co-authored-by: Codex <codex@openai.com>
Document the retained snapshot task, deferred row-reader callback, and fresh task streams returned by repeated planning.

Co-authored-by: Codex <codex@openai.com>
Validate common metadata and schema inputs through MetadataScanContext::Validate at builder and scan factory boundaries. Keep FileIO validation specific to snapshot scans and cover direct factory input validation and static scans without FileIO.

Co-authored-by: Codex <codex@openai.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants