Preserve partitioning on temp FILE_LOADS tables

[PRADDZY]

Summary

• preserve destination partitioning metadata when FILE_LOADS writes through temporary tables
• reuse the destination table lookup for missing schema resolution and only inject partitioning when the caller did not already supply it
• add regression coverage for inherited time/range partitioning and explicit partitioning precedence

Fixes #38017.

Testing

• python -m pytest apache_beam/io/gcp/bigquery_file_loads_test.py -k "temporary_table_load_inherits_destination_time_partitioning or temporary_table_load_inherits_destination_range_partitioning or temporary_table_load_keeps_explicit_partitioning_parameters" -q
• python -m pytest apache_beam/io/gcp/bigquery_file_loads_test.py::TestBigQueryFileLoads::test_wait_for_load_job_completion -q
• python -m py_compile apache_beam/io/gcp/bigquery_file_loads.py apache_beam/io/gcp/bigquery_file_loads_test.py

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds support for temporary-table BigQuery file loads to inherit partitioning settings from the final destination table when not explicitly provided, and validates this behavior with new unit tests.

Changes:

• Add helpers to detect/propagate timePartitioning and rangePartitioning into load job parameters for temporary table loads.
• Update TriggerLoadJobs.process to optionally fetch the destination table once to reuse schema and partitioning metadata.
• Add tests verifying partitioning inheritance and that explicit partitioning parameters are not overridden.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
sdks/python/apache_beam/io/gcp/bigquery_file_loads.py	Propagates destination-table partitioning settings into temporary-table load job parameters and reuses fetched table metadata.
sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py	Adds unit tests for partitioning inheritance and explicit-parameter precedence.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request improves the reliability of BigQuery file load operations by ensuring that partitioning settings are correctly propagated to temporary tables. By intelligently fetching destination table metadata and avoiding redundant lookups, the changes ensure that existing partitioning configurations are respected unless overridden by the user. This fix addresses issues where temporary table creation might otherwise lose critical partitioning constraints.

Highlights

• Partitioning Preservation: Ensures that BigQuery destination partitioning metadata (time and range) is correctly preserved when writing through temporary tables in FILE_LOADS.
• Smart Lookup Logic: Optimizes destination table lookups by reusing existing schema resolution logic and only injecting partitioning parameters when they are not explicitly provided by the caller.
• Regression Testing: Added comprehensive test cases to verify inheritance of time/range partitioning and to ensure explicit partitioning parameters take precedence.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request ensures that temporary table loads inherit time and range partitioning parameters from the destination BigQuery table. It introduces helper functions to check and apply these parameters, updates the TriggerLoadJobs process to fetch the destination table configuration, and adds corresponding unit tests. However, the current implementation introduces a performance regression by bypassing the schema cache and making redundant, synchronous get_table API calls on every bundle. It is recommended to cache the fetched destination table locally to avoid unnecessary API calls.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[github-actions]

Assigning reviewers:

R: @shunping for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[github-actions]

Reminder, please take a look at this pr: @shunping

[github-actions]

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @tvalentyn for label python.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

[github-actions]

Reminder, please take a look at this pr: @tvalentyn

[github-actions]

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @claudevdm for label python.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

[tvalentyn]

/gemini review

[tvalentyn]

cc: @ahmedabu98 in case you looked at this issue already and would like to chime in. Thanks

[gemini-code-assist]

Code Review

This pull request updates Apache Beam's BigQuery file loads to inherit partitioning parameters (time or range partitioning) from the destination table when creating temporary tables. It also introduces caching for the destination table to avoid redundant API calls. Feedback on the changes points out a potential performance bottleneck: if get_table raises an exception, the failure is not cached, which could lead to repeated API calls for subsequent partitions in the same bundle. It is recommended to cache None in self.destination_table_cache when an exception occurs.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[ahmedabu98]

I think this looks great.

Can we add an integration test to make sure it works against the real API?

[github-actions]

Reminder, please take a look at this pr: @claudevdm

[tvalentyn]

waiting on author

[tvalentyn]

hi @PRADDZY -- could you please address the open comments from @ahmedabu98 ? Thanks!

[PRADDZY]

Thanks @ahmedabu98. Added a BigQuery write IT in bigquery_write_it_test.py that pre-creates a date-partitioned destination table, forces the FILE_LOADS temp-table/copy path, verifies the rows through BigqueryFullResultMatcher, and verifies the destination timePartitioning through BigQueryTableMatcher. I also addressed the inline comment by removing the stale schema-conversion try/except around the non-API call.