TRT-2811: Make prow loader batch writes resilient to single-batch failures

[mstaeble]

Summary

• When a batch write fails in accumulateAndWriteJobRuns, log the error and continue processing subsequent batches instead of aborting the entire load
• Each batch error is appended individually to pl.errors so LoaderWithMetrics counts them separately in Prometheus metrics
• A ctx.Err() check at the top of each loop iteration exits cleanly on cancellation to avoid doomed write attempts
• Removed cancelFetch() call since loadDailyTestAnalysisByJob sources from BigQuery and doesn't depend on PG writes

Test plan

• go vet ./pkg/dataloader/prowloader/... passes
• go test ./pkg/dataloader/prowloader/... passes
• make lint passes

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Improved resilience when processing job-run batches.
  • Processing now continues after individual batch write failures, while failures are recorded for reporting.
  • Processing stops promptly when cancellation is requested.
  • Completion metrics now reflect the number of successfully processed job runs.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[openshift-ci-robot]

@mstaeble: This pull request references TRT-2811 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

• When a batch write fails in accumulateAndWriteJobRuns, log the error and continue processing subsequent batches instead of aborting the entire load
• Each batch error is appended individually to pl.errors so LoaderWithMetrics counts them separately in Prometheus metrics
• A ctx.Err() check at the top of each loop iteration exits cleanly on cancellation to avoid doomed write attempts
• Removed cancelFetch() call since loadDailyTestAnalysisByJob sources from BigQuery and doesn't depend on PG writes

Test plan

• go vet ./pkg/dataloader/prowloader/... passes
• go test ./pkg/dataloader/prowloader/... passes
• make lint passes
• Monitor loader logs in staging/prod for batch write errors; confirm subsequent batches continue processing after a failure
• Verify error count metric reflects individual batch failures (not collapsed into 1)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

Walkthrough

Prow job-run loading now performs best-effort batch writes, records batch failures, continues processing after write errors, respects context cancellation, and updates processed metrics from successful results.

Changes

Prow job-run processing

Layer / File(s)	Summary
Best-effort batch writes pkg/dataloader/prowloader/prow.go	accumulateAndWriteJobRuns continues flushing batches after write failures, records errors and counters, stops on context cancellation, and updates the processed metric. Load no longer propagates batch-write errors through cancellation and returned errors.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: neisw, sosiouxme

🚥 Pre-merge checks | ✅ 20 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Test Coverage For New Features	⚠️ Warning	The PR changes prow.go only; no tests were added or updated, and prowloader tests don't cover the new batch-write/cancellation behavior.	Add regression unit tests for failed-then-successful batch writes, per-batch pl.errors accounting, and cancellation before the final flush (use injectable batch writer).

✅ Passed checks (20 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Go Error Handling	✅ Passed	New batch-write errors are wrapped with %w, no new bare _ ignores or panics were added, and no new nil-deref risk appears in the changed path.
Sql Injection Prevention	✅ Passed	The patch only changes batch error handling; SQL in writeJobRunBatch stays static, and temp-table names are validated before use.
Excessive Css In React Should Use Styles	✅ Passed	The PR only changes a Go backend loader; no React components, inline CSS, or style objects are present, so the CSS/useStyles rule is not applicable.
Single Responsibility And Clear Naming	✅ Passed	PASS: The PR keeps the cohesive prowloader package focused, and the new methods (accumulateAndWriteJobRuns, writeJobRunBatch) are specific and action-oriented.
Feature Documentation	✅ Passed	Only pkg/dataloader/prowloader/prow.go changed; docs/features/job-analysis-symptoms.md documents label ingestion, not the internal batch-write control flow, so no doc update is needed.
Stable And Deterministic Test Names	✅ Passed	Only pkg/dataloader/prowloader/prow.go changed; no test titles were added or modified, so the naming rule is unaffected.
Test Structure And Quality	✅ Passed	No Ginkgo test code was added or modified in this PR; only pkg/dataloader/prowloader/prow.go changed, so the test-quality check is not applicable.
Microshift Test Compatibility	✅ Passed	Only pkg/dataloader/prowloader/prow.go changed; no new Ginkgo e2e tests or MicroShift-sensitive APIs were added.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; the PR only changes pkg/dataloader/prowloader/prow.go, so SNO-specific test compatibility is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	The touched code is loader/batch-processing only; no manifests, controllers, or scheduling constraints were added.
Ote Binary Stdout Contract	✅ Passed	PASS: The PR only changes pkg/dataloader/prowloader/prow.go, a library file; it adds no main/init/TestMain setup or stdout-printing calls, and no klog/stdout writes were found.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	Only pkg/dataloader/prowloader/prow.go changed; no new Ginkgo e2e tests or network-facing test code were added, so the IPv6/disconnected-network check is not applicable.
No-Weak-Crypto	✅ Passed	Only prowloader batch-write control flow changed; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons were added.
Container-Privileges	✅ Passed	Only pkg/dataloader/prowloader/prow.go changed; no container/K8s manifests or privileged/securityContext settings were touched.
No-Sensitive-Data-In-Logs	✅ Passed	New logs only include constant messages and batch/result counts; no passwords, tokens, PII, hostnames, or customer data are logged.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: making prow loader batch writes tolerate single-batch failures.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mstaeble]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 50 minutes.

[mstaeble]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/dataloader/prowloader/prow.go`:
- Around line 1206-1246: The accumulateAndWriteJobRuns flow currently hard-codes
writeJobRunBatch, preventing isolated regression tests. Add a function-type
writer field to ProwLoader, initialize it to writeJobRunBatch in production, and
have accumulateAndWriteJobRuns invoke that field; then add prow_test.go coverage
for failed-then-successful batches, per-batch pl.errors, and cancellation with a
pending batch.
- Around line 1225-1235: Update the result-processing loop around batch
accumulation and the final flush so cancellation is honored: check ctx.Err()
before appending each result and guard the len(batch) > 0 final flush with
ctx.Err() == nil. Preserve normal threshold-based flushing when the context
remains active.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 2b1f43d8-f50f-4476-945f-9b7e3a6bac25

📥 Commits

Reviewing files that changed from the base of the PR and between de6d177 and a101186.

📒 Files selected for processing (1)

• pkg/dataloader/prowloader/prow.go

[openshift-merge-bot]

Scheduling required tests:
/test e2e

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mstaeble, petr-muller

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [mstaeble,petr-muller]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-merge-bot]

Scheduling required tests:
/test e2e

[openshift-ci]

@mstaeble: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.