Skip to content

CNTRLPLANE-3919: data race in TestEnqueueNodePoolsForCloudConfig - #9095

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
sdminonne:CNTRLPLANE-3919
Jul 25, 2026
Merged

CNTRLPLANE-3919: data race in TestEnqueueNodePoolsForCloudConfig#9095
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
sdminonne:CNTRLPLANE-3919

Conversation

@sdminonne

@sdminonne sdminonne commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • TestEnqueueNodePoolsForCloudConfig parallel subtests share mutable HostedControlPlane and NodePool object pointers. fake.ClientBuilder.Build() mutates these via SetResourceVersion(), causing a data race that fails the entire nodepool test package intermittently in ci/prow/unit.
  • Deep-copy shared objects before passing them to WithObjects() so each parallel subtest operates on its own copy.

Test plan

  • go test -race -run TestEnqueueNodePoolsForCloudConfig -count=5 ./hypershift-operator/controllers/nodepool/ passes clean
  • Full nodepool controller unit test suite passes with race detector

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Improved test isolation for parallel node pool controller tests.
    • Prevented shared test data from being mutated across concurrent test cases.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: e4db76fa-a834-447f-99c1-6f7735255848

📥 Commits

Reviewing files that changed from the base of the PR and between 144cca9 and 74f889d.

📒 Files selected for processing (1)
  • hypershift-operator/controllers/nodepool/nodepool_controller_test.go

📝 Walkthrough

Walkthrough

TestEnqueueNodePoolsForCloudConfig now deep-copies each test case object into a local slice before creating the fake Kubernetes client. The client uses api.Scheme and the copied objects, preventing shared object state across parallel subtests.

Suggested reviewers: devguyio, csrwng, bryan-cox

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: fixing a data race in TestEnqueueNodePoolsForCloudConfig.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR only deep-copies objects in an existing table-driven test; no Ginkgo titles were added or modified, and the visible test names are static strings.
Test Structure And Quality ✅ Passed The only changed test is a pure unit table test; it deep-copies shared objects, has one-behavior subtests, and uses no cluster waits or leaked resources.
Topology-Aware Scheduling Compatibility ✅ Passed Only a test fixture change: deep-copying objects before fake client setup; no manifests, controllers, or scheduling logic/topology assumptions were added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR only changes a unit test using t.Run/fake client deep-copies; no new Ginkgo e2e tests, IPv4-only assumptions, or external connectivity were added.
No-Weak-Crypto ✅ Passed PR only deep-copies test objects in a nodepool test; no weak crypto, custom crypto, or secret-comparison changes were added.
Container-Privileges ✅ Passed PR only deep-copies test objects in nodepool_controller_test.go; no manifests or securityContext/privilege settings were added or changed.
No-Sensitive-Data-In-Logs ✅ Passed The change only deep-copies test objects before fake client creation; no new logs or sensitive data exposure were added.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release area/platform/azure PR/issue for Azure (AzurePlatform) platform labels Jul 24, 2026
@openshift-ci
openshift-ci Bot requested review from csrwng and devguyio July 24, 2026 07:16
@openshift-ci openshift-ci Bot added area/testing Indicates the PR includes changes for e2e testing and removed do-not-merge/needs-area labels Jul 24, 2026
@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.51%. Comparing base (144cca9) to head (74f889d).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9095   +/-   ##
=======================================
  Coverage   44.51%   44.51%           
=======================================
  Files         774      774           
  Lines       96997    96997           
=======================================
  Hits        43179    43179           
  Misses      50830    50830           
  Partials     2988     2988           
Flag Coverage Δ
cmd-support 38.39% <ø> (ø)
cpo-hostedcontrolplane 47.22% <ø> (ø)
cpo-other 45.25% <ø> (ø)
hypershift-operator 54.45% <ø> (ø)
other 32.64% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
test/e2e/v2/tests/nodepool_osimagestream_test.go (1)

58-78: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate OSStreams feature-gate CRD check across two files. Both sites independently re-implement the same "does the CRD expose spec.osImageStream/status.osImageStream" check via e2eutil.HasFieldInCRDSchema, which will drift if the check logic changes.

  • test/e2e/v2/tests/nodepool_osimagestream_test.go#L58-L78: extract this dual HasFieldInCRDSchema check (spec + status) into a shared exported helper, e.g. e2eutil.IsOSImageStreamFeatureEnabled(ctx, client) (bool, error), and call it here.
  • test/e2e/v2/tests/nodepool_lifecycle_test.go#L1466-L1509: replace the inline duplicate check in verifyOSImageStreamAfterUpgrade with the same shared helper.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/v2/tests/nodepool_osimagestream_test.go` around lines 58 - 78,
Extract the duplicated spec.osImageStream and status.osImageStream CRD checks
into an exported e2eutil.IsOSImageStreamFeatureEnabled(ctx, client) helper
returning (bool, error). Update osImageStreamBeforeEach in
test/e2e/v2/tests/nodepool_osimagestream_test.go#L58-L78 and
verifyOSImageStreamAfterUpgrade in
test/e2e/v2/tests/nodepool_lifecycle_test.go#L1466-L1509 to use the helper,
preserving each caller’s existing error handling and skip behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Makefile`:
- Around line 106-114: Update PULL_BASE_SHA and the api-lint/api-lint-fix
targets so the base revision is validated before invoking golangci-lint. Resolve
a verified available ref using appropriate fallbacks such as origin/main, and
fail explicitly with a diagnostic when no suitable base ref exists instead of
passing an empty --new-from-rev value.

In `@test/e2e/v2/tests/nodepool_osimagestream_test.go`:
- Around line 356-426: Update NodePoolOSImageStreamExplicitDefaultNoRolloutTest
to avoid patching the live default NodePool: clone its relevant configuration
into an isolated zero-replica NodePool using buildTestNodePool, then apply the
explicit version-derived osImageStream to that test resource and assert its
config hash remains unchanged. Preserve the existing default-stream selection
and no-rollout verification while removing cleanup that mutates the shared
default NodePool.

---

Nitpick comments:
In `@test/e2e/v2/tests/nodepool_osimagestream_test.go`:
- Around line 58-78: Extract the duplicated spec.osImageStream and
status.osImageStream CRD checks into an exported
e2eutil.IsOSImageStreamFeatureEnabled(ctx, client) helper returning (bool,
error). Update osImageStreamBeforeEach in
test/e2e/v2/tests/nodepool_osimagestream_test.go#L58-L78 and
verifyOSImageStreamAfterUpgrade in
test/e2e/v2/tests/nodepool_lifecycle_test.go#L1466-L1509 to use the helper,
preserving each caller’s existing error handling and skip behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: d395c690-68f5-40bf-9baf-7e2a327e65cb

📥 Commits

Reviewing files that changed from the base of the PR and between 144cca9 and df76eba.

⛔ Files ignored due to path filters (12)
  • control-plane-operator/controllers/hostedcontrolplane/testdata/ingress-operator/AROSwift/zz_fixture_TestControlPlaneComponents_ingress_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/ingress-operator/GCP/zz_fixture_TestControlPlaneComponents_ingress_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/ingress-operator/IBMCloud/zz_fixture_TestControlPlaneComponents_ingress_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/ingress-operator/ModernTLS/zz_fixture_TestControlPlaneComponents_ingress_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/ingress-operator/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_ingress_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/ingress-operator/zz_fixture_TestControlPlaneComponents_ingress_operator_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/AROSwift/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/GCP/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/IBMCloud/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/ModernTLS/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/TechPreviewNoUpgrade/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
  • control-plane-operator/controllers/hostedcontrolplane/testdata/kube-apiserver/zz_fixture_TestControlPlaneComponents_kube_apiserver_deployment.yaml is excluded by !**/testdata/**
📒 Files selected for processing (12)
  • Makefile
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/assets_test.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/ingress-operator/deployment.yaml
  • control-plane-operator/controllers/hostedcontrolplane/v2/assets/kube-apiserver/deployment.yaml
  • control-plane-operator/controllers/hostedcontrolplane/v2/ingressoperator/deployment.go
  • hypershift-operator/controllers/nodepool/nodepool_controller_test.go
  • hypershift-operator/controllers/nodepool/version.go
  • hypershift-operator/controllers/nodepool/version_test.go
  • test/e2e/util/eventually.go
  • test/e2e/v2/lifecycle/azure.go
  • test/e2e/v2/tests/nodepool_lifecycle_test.go
  • test/e2e/v2/tests/nodepool_osimagestream_test.go

Comment thread Makefile
Comment on lines +106 to +114
PULL_BASE_SHA ?= $(if $(UPSTREAM_REMOTE),$(shell git rev-parse $(UPSTREAM_REMOTE)/main), $(shell git rev-parse main))

.PHONY: api-lint
api-lint: $(GOLANGCI_LINT) $(KUBEAPILINTER_PLUGIN)
cd api && $(GOLANGCI_LINT) run --config ./.golangci.yml --modules-download-mode=readonly -v $(if $(PULL_BASE_SHA),--new-from-rev=$(PULL_BASE_SHA) --whole-files)
cd api && $(GOLANGCI_LINT) run --config ./.golangci.yml --modules-download-mode=readonly -v --new-from-rev=${PULL_BASE_SHA}

.PHONY: api-lint-fix
api-lint-fix: $(GOLANGCI_LINT) $(KUBEAPILINTER_PLUGIN)
cd api && $(GOLANGCI_LINT) run --config ./.golangci.yml --fix -v $(if $(PULL_BASE_SHA),--new-from-rev=$(PULL_BASE_SHA) --whole-files)
cd api && $(GOLANGCI_LINT) run --config ./.golangci.yml --fix -v --new-from-rev=${PULL_BASE_SHA}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Ensure PULL_BASE_SHA resolves before invoking the linter.

git rev-parse $(UPSTREAM_REMOTE)/main and git rev-parse main are unguarded. In a checkout with only origin/main, or without a fetched local/upstream main, the variable becomes empty and both targets pass --new-from-rev=, causing lint to fail. Use verified fallbacks such as origin/main, or fail explicitly with a diagnostic before invoking golangci-lint.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Makefile` around lines 106 - 114, Update PULL_BASE_SHA and the
api-lint/api-lint-fix targets so the base revision is validated before invoking
golangci-lint. Resolve a verified available ref using appropriate fallbacks such
as origin/main, and fail explicitly with a diagnostic when no suitable base ref
exists instead of passing an empty --new-from-rev value.

Comment thread test/e2e/v2/tests/nodepool_osimagestream_test.go
@jparrill

jparrill commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

/retitle CNTRLPLANE-3919: data race in TestEnqueueNodePoolsForCloudConfig

@openshift-ci openshift-ci Bot changed the title fix(CNTRLPLANE-3919): data race in TestEnqueueNodePoolsForCloudConfig CNTRLPLANE-3919: data race in TestEnqueueNodePoolsForCloudConfig Jul 24, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 24, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 24, 2026

Copy link
Copy Markdown

@sdminonne: This pull request references CNTRLPLANE-3919 which is a valid jira issue.

Details

In response to this:

Summary

  • TestEnqueueNodePoolsForCloudConfig parallel subtests share mutable HostedControlPlane and NodePool object pointers. fake.ClientBuilder.Build() mutates these via SetResourceVersion(), causing a data race that fails the entire nodepool test package intermittently in ci/prow/unit.
  • Deep-copy shared objects before passing them to WithObjects() so each parallel subtest operates on its own copy.

Test plan

  • go test -race -run TestEnqueueNodePoolsForCloudConfig -count=5 ./hypershift-operator/controllers/nodepool/ passes clean
  • Full nodepool controller unit test suite passes with race detector

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

  • Added NodePool OS image stream validation and status reporting, including RHEL 9 and RHEL 10 support.

  • Added post-upgrade verification for reported OS image streams.

  • Added HAProxy image configuration options for ingress components.

  • Configured Konnectivity Server to use the cluster CA certificate.

  • Bug Fixes

  • Improved compatibility with newer operating system image version formats.

  • Prevented unnecessary NodePool rollouts when explicitly setting the derived default OS image stream.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

…udConfig

Parallel subtests share the same HostedControlPlane and NodePool
pointers. fake.ClientBuilder.Build() mutates these objects via
SetResourceVersion(), causing a data race detected by -race.

Deep-copy each object before passing it to WithObjects() so each
subtest operates on its own copy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@sdminonne

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@bryan-cox bryan-cox left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@openshift-ci

openshift-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, sdminonne

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 24, 2026
@jparrill

Copy link
Copy Markdown
Contributor

/lgtm

@jparrill

Copy link
Copy Markdown
Contributor

/verified by unit tests

Being a fix for a unit test, don't need more explanation.

@jparrill

Copy link
Copy Markdown
Contributor

/label acknowledge-critical-fixes-only

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 24, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@jparrill: This PR has been marked as verified by unit tests.

Details

In response to this:

/verified by unit tests

Being a fix for a unit test, don't need more explanation.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Jul 24, 2026
@bryan-cox

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 24, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test unit
/test verify

@bryan-cox

Copy link
Copy Markdown
Member

/retest

1 similar comment
@bryan-cox

Copy link
Copy Markdown
Member

/retest

@bryan-cox

Copy link
Copy Markdown
Member

/test e2e-aks-4-22
/test e2e-aws-4-22

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 144cca9 and 2 for PR HEAD 74f889d in total

@sdminonne

Copy link
Copy Markdown
Contributor Author

/retest-required

1 similar comment
@bryan-cox

Copy link
Copy Markdown
Member

/retest-required

@sdminonne

Copy link
Copy Markdown
Contributor Author

/test e2e-aws-4-22

@sdminonne

Copy link
Copy Markdown
Contributor Author

/test e2e-aks-4-22

@openshift-ci

openshift-ci Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

@sdminonne: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit d166b4f into openshift:main Jul 25, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release area/platform/azure PR/issue for Azure (AzurePlatform) platform area/testing Indicates the PR includes changes for e2e testing jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants