[Data Validation] Add table configuration by aasthabharill · Pull Request #4188 · GoogleCloudPlatform/DataflowTemplates

aasthabharill · 2026-08-28T12:27:38Z

b/514940857

Added two new pipeline parameters:

tables: comma-separated list of source tables for small configurations
tableListFilePath: GCS File path containing list of source tables in a text file for large workloads

Design Decisions

Why two different template params?

tables will be extremely inconvenient to configure in case user needs to configure large number of tables

`tableListFilePath` is JSON file

Extensible for future additions
Example:

{
  "tableNames": [
    "Users",
    "AccountRoles"
  ]
}

Table names are to be specified as per their source names

More intuitive for users as they are more familiar with their source schemas. In case of Spanner table renames, the user specifies an overrides file/session file - which will be used to map their configured tables to their corresponding Spanner names

Fail fast if both template params are provided

There should be only one source of truth for the configured tables and since the purpose for both template params is the same, it doesn't make sense to pass both.

No mechanism to fail fast if configured tables do not exist in source

Since our source is the GCS Avro directory, if a source table exists but has zero rows, it will not have any entry in GCS.
Therefore, GCS records for a table not existing could mean either of two things:

The table doesn't exist on source and was misconfigured by the user
The table does exist on source but has 0 rows.
Since there's no way to differentiate between these two cases, we cannot introduce a fail-fast loop to identify a misconfigured table list.

Testing

Added comprehensive unit tests for all changes
Added an integration test for the scenario where tables mentions a subset of tables which were renamed in spanner (so tests table config in combination with overrides)

gemini-code-assist · 2026-09-02T11:07:47Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces table-level filtering for the GCS-to-Spanner Data Validation pipeline. By allowing users to specify a subset of tables via a comma-separated list or a GCS file path, the pipeline can now perform targeted validation, significantly improving efficiency when working with large schemas.

Highlights

Table Filtering Configuration: Added new template parameters 'tables' and 'tableListFilePath' to allow users to specify a subset of tables for validation.
Pipeline Integration: Implemented a new 'TableSelectionConfig' class and integrated it into the source and Spanner reader transforms to filter data processing.
Testing and Documentation: Updated documentation and added comprehensive unit and integration tests to ensure the new filtering logic functions correctly.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces table-based filtering to the GCS to Spanner Data Validation pipeline, allowing users to specify a subset of tables to validate via a comma-separated list or a GCS file path. Feedback on the changes focuses on improving robustness and code quality, including adding defensive null checks to prevent potential NullPointerExceptions in SourceReaderTransform, TableSelectionConfig, and CreateSpannerReadOpsFn. Additionally, it is recommended to use StandardCharsets.UTF_8 instead of string literals, and to remove a redundant Thread.sleep call in the integration tests as the Dataflow runner's startup time is already sufficient to satisfy the staleness window.

codecov · 2026-09-02T11:19:54Z

Codecov Report

❌ Patch coverage is 86.74699% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.20%. Comparing base (9d1e211) to head (9892048).

Files with missing lines	Patch %	Lines
...e/cloud/teleport/v2/config/TableConfiguration.java	87.23%	0 Missing and 6 partials ⚠️
...oud/teleport/v2/config/TableConfigurationFile.java	25.00%	3 Missing ⚠️
...cloud/teleport/v2/dofn/CreateSpannerReadOpsFn.java	91.66%	0 Missing and 1 partial ⚠️
...ogle/cloud/teleport/v2/templates/GCSSpannerDV.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #4188       +/-   ##
=============================================
+ Coverage     33.88%   63.20%   +29.31%     
- Complexity      503     2816     +2313     
=============================================
  Files           215      564      +349     
  Lines         13133    32721    +19588     
  Branches       1304     3671     +2367     
=============================================
+ Hits           4450    20681    +16231     
- Misses         8318    10987     +2669     
- Partials        365     1053      +688

Components	Coverage Δ
spanner-templates	`84.70% <86.74%> (∅)`
spanner-import-export	`∅ <ø> (∅)`
spanner-live-forward-migration	`88.66% <ø> (∅)`
spanner-live-reverse-replication	`81.37% <ø> (∅)`
spanner-bulk-migration	`89.06% <ø> (∅)`
gcs-spanner-dv	`87.86% <86.74%> (∅)`

Files with missing lines	Coverage Δ
.../teleport/v2/transforms/SourceReaderTransform.java	`100.00% <100.00%> (ø)`
...teleport/v2/transforms/SpannerReaderTransform.java	`100.00% <100.00%> (ø)`
...cloud/teleport/v2/dofn/CreateSpannerReadOpsFn.java	`94.11% <91.66%> (ø)`
...ogle/cloud/teleport/v2/templates/GCSSpannerDV.java	`2.43% <0.00%> (ø)`
...oud/teleport/v2/config/TableConfigurationFile.java	`25.00% <25.00%> (ø)`
...e/cloud/teleport/v2/config/TableConfiguration.java	`87.23% <87.23%> (ø)`

... and 439 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

manitgupta · 2026-09-03T03:20:58Z

tableListFilePath is flat text file
Very easy to make, maintain and edit for customers as it's just a list of tables. JSON for this would be overkill and would require formatters for no real advantage.

I don't think it should be flat text file, I think it should be a JSON. Reason - extensibility.
A TableConfiguration can have more aspects about a table such as which columns to include/exclude etc. and keeping a JSON (backed by a POJO in the code) will keep this configuration flexible and extendible.

manitgupta

I haven't looked at the full PR yet, lets align on the existing comments and then will look at the rest

manitgupta · 2026-09-03T03:22:59Z

+ * Configuration class for table-based filtering in Data Validation pipeline. Encapsulates parsing,
+ * matching, and validation of source and Spanner tables.
+ */
+public class TableSelectionConfig implements Serializable {


I think this should be a named TableConfiguration (mentioned the reason in a different comment - extensibility).

The TableConfiguration should have a field tableName which is used for filtering.

will make these changes + JSON and get back

manitgupta · 2026-09-03T10:03:18Z

Can you put a sample tableConfiguration JSON in the description? I would like to have a look once

pull-request-size Bot added the size/L label Aug 28, 2026

aasthabharill added the addition New feature or request label Aug 28, 2026

pull-request-size Bot added size/XL size/XXL and removed size/L size/XL labels Sep 1, 2026

aasthabharill marked this pull request as ready for review September 2, 2026 11:04

aasthabharill requested a review from a team as a code owner September 2, 2026 11:04

aasthabharill requested review from jsuhani-2026 and shreyakhajanchi September 2, 2026 11:04

gemini-code-assist Bot reviewed Sep 2, 2026

View reviewed changes

aasthabharill requested a review from manitgupta September 2, 2026 11:42

manitgupta requested changes Sep 3, 2026

View reviewed changes

aasthabharill added 14 commits September 3, 2026 12:49

initial working changes

8993921

draft 2

fe4c7c4

testing

6ad5f6a

clean

f3d9f70

TableSelectionConfig

81b0ab4

final changes

95eb279

gemini-review

ce1ebfe

codecov

4df885a

it change

8cafb62

it change

5c07e13

it change

c323e79

json changes+rename

8f0b959

remove readme changes

6956647

terraform + rename

9323e2f

aasthabharill force-pushed the dv-table-config branch from 92afbd3 to 9323e2f Compare September 3, 2026 08:58

final touches

9892048

Navigation Menu

Uh oh!

[Data Validation] Add table configuration - #4188

[Data Validation] Add table configuration#4188
aasthabharill wants to merge 15 commits into
mainfrom
dv-table-config

aasthabharill commented Aug 28, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Sep 2, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

manitgupta commented Sep 3, 2026

Uh oh!

manitgupta left a comment

Uh oh!

manitgupta Sep 3, 2026

Uh oh!

aasthabharill Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

manitgupta commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aasthabharill commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Design Decisions

Why two different template params?

tableListFilePath is JSON file

Table names are to be specified as per their source names

Fail fast if both template params are provided

No mechanism to fail fast if configured tables do not exist in source

Testing

Uh oh!

gemini-code-assist Bot commented Sep 2, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

manitgupta commented Sep 3, 2026

Uh oh!

manitgupta left a comment

Choose a reason for hiding this comment

Uh oh!

manitgupta Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

aasthabharill Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

manitgupta commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aasthabharill commented Aug 28, 2026 •

edited

Loading

`tableListFilePath` is JSON file

codecov Bot commented Sep 2, 2026 •

edited

Loading