Skip to content

OCPBUGS-99272: Auto-remove capi-provider resource limits with user-override detection#9047

Draft
sdminonne wants to merge 1 commit into
openshift:mainfrom
sdminonne:OCPBUGS-99272
Draft

OCPBUGS-99272: Auto-remove capi-provider resource limits with user-override detection#9047
sdminonne wants to merge 1 commit into
openshift:mainfrom
sdminonne:OCPBUGS-99272

Conversation

@sdminonne

@sdminonne sdminonne commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Fix CPOv2 framework unconditionally preserving container resource limits on the capi-provider deployment during reconciliation, which prevented manifest-driven limit removal from taking effect on upgrade
  • Scoped to capi-provider only — all other components retain the existing behavior of unconditionally preserving existing resources
  • When the manifest no longer specifies limits but the live capi-provider deployment has them, automatically remove the limits and record the action via a limits-removed annotation
  • If a user manually re-adds limits to a container already listed in the annotation, the operator detects this as an intentional override and preserves the user's resources
  • Add a container-resources-hash annotation on the pod template so DeepDerivative detects limit removal even when the new resource spec is a subset of the old one

Test plan

  • Unit tests for limitsAlreadyRemoved annotation parsing
  • Unit tests for markLimitsRemoved annotation writing (including deduplication)
  • Unit tests for computeContainerResourcesHash stability and order independence
  • Unit test: capi-provider manifest has no limits, existing has limits, no annotation → limits removed, annotation set
  • Unit test: capi-provider manifest has no limits, existing has limits, annotation set → user override preserved
  • Unit test: capi-provider manifest has no limits, existing has no limits → existing resources preserved
  • Unit test: capi-provider manifest has limits, existing has limits → existing resources preserved
  • Unit test: non-capi-provider component with no manifest limits and existing limits → existing resources always preserved
  • Unit test: no existing deployment → manifest resources used as-is
  • Unit test: container-resources-hash annotation is set on pod template
  • e2e-aws-upgrade-hypershift-operator — validates the rollout is safe

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved workload reconciliation to consider existing workload annotations when applying default options.
    • Updated capi-provider container resource reconciliation: when manifests omit limits, the operator now removes limits as needed while honoring prior user re-adds.
    • Added stable, order-independent hashing of per-container resource requirements to ensure changes are reliably detected and rolled out.
  • Tests
    • Updated unit tests for the revised setDefaultOptions call signature.
    • Added coverage for limit-removal bookkeeping and deterministic resource hashing behavior.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 21, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@sdminonne: This pull request references Jira Issue OCPBUGS-99272, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

  • The CPOv2 framework unconditionally preserves existing container resource requirements (requests/limits) during reconciliation. This prevents manifest-driven changes from taking effect on upgrade. For example, OCPBUGS-76997 removed resource limits from the capi-provider deployment, but the fix is never applied on upgrade because the old limits are restored every reconciliation loop.
  • A secondary issue compounds this: DeepDerivative ignores nil/absent fields, so even without the preservation code, removing limits (setting to nil) wouldn't trigger an update.
  • This PR flips the default so the operator reconciles container resources to whatever the manifest says. Users who have manually tuned resources can opt in to preserving them via a hypershift.openshift.io/preserve-container-resources annotation on the HostedControlPlane (comma-separated deployment names or * for all).
  • A resource-hash annotation on pod templates ensures DeepDerivative detects resource changes including removals, using the same pattern as the existing config-hash.

How it fixes OCPBUGS-99272

Repro: install a pre-fix MCE (e.g., 2.10.3) so capi-provider has limits, then upgrade to a post-fix MCE (e.g., 2.11.3) whose manifest removes limits per OCPBUGS-76997.

Before this fix: the framework copies the old limits from the live deployment back onto the desired state every reconciliation, so the upgrade never removes them. The only workaround is to delete the deployment and let it be recreated.

After this fix: no preserve-container-resources annotation is set (default), so the operator uses the manifest as-is (no limits). The resource hash changes because "limits present" and "limits absent" produce different hashes, forcing DeepDerivative to trigger the update. The limits are removed automatically — no manual intervention needed.

The annotation

The hypershift.openshift.io/preserve-container-resources annotation is for users who have manually tuned container resources (e.g., increased requests for a large cluster) and don't want the operator to overwrite them. It applies on every reconciliation loop, not just upgrades. Without the annotation, the manifest always wins.

Test plan

  • go test ./support/controlplane-component/... — framework unit tests
  • make verify — linting and generation
  • Verify the OCPBUGS-99272 repro scenario: upgrade from pre-fix to post-fix MCE and confirm limits are removed without deleting the deployment
  • Verify annotation opt-in: set hypershift.openshift.io/preserve-container-resources=capi-provider on HCP, manually edit resources, confirm they survive reconciliation

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The workload reconciler now passes existing workload annotations into setDefaultOptions. Container resource handling supports first-time limit removal for capi-provider, preserves later user re-additions, and retains existing resources in other scenarios. The pod template records removed-limit containers and a stable hash of final container resources. Tests cover annotation bookkeeping, hashing, updated calls, and reconciliation scenarios.

Sequence Diagram(s)

sequenceDiagram
  participant WorkloadReconciler
  participant ExistingWorkload
  participant DefaultOptions
  participant PodTemplate
  WorkloadReconciler->>ExistingWorkload: read annotations
  WorkloadReconciler->>DefaultOptions: pass annotations
  DefaultOptions->>PodTemplate: reconcile container resources
  DefaultOptions->>PodTemplate: write removal tracking and resource hash
Loading

Suggested reviewers: ironcladlou, devguyio

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: auto-removing capi-provider limits and preserving user overrides.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The changed tests use static, deterministic names; no It/Describe/Context/When titles interpolate dynamic values or random identifiers.
Test Structure And Quality ✅ Passed The new tests are table-driven unit tests, each subtest targets one scenario; no cluster resources or waits are introduced, and the style matches existing package tests.
Topology-Aware Scheduling Compatibility ✅ Passed Only resource-limit defaulting/annotations changed; no affinity, spread, nodeSelector, replica, or topology assumptions were added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No new Ginkgo e2e tests or external/network assumptions were added; the diff only touches unit tests under support/controlplane-component.
No-Weak-Crypto ✅ Passed Only SHA-256 is added for a non-security resource hash; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons were introduced.
Container-Privileges ✅ Passed Touched files only change Go reconciliation logic/tests; no privileged, hostNetwork/IPC/PID, SYS_ADMIN, root, or allowPrivilegeEscalation settings were added.
No-Sensitive-Data-In-Logs ✅ Passed No new logging statements were added in the changed files; the diff only adjusts reconciliation logic, annotations, and tests.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@sdminonne
sdminonne marked this pull request as draft July 21, 2026 16:10
@openshift-ci openshift-ci Bot added do-not-merge/needs-area do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jul 21, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
support/controlplane-component/defaults_test.go (1)

691-802: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use the required test-case description format.

  • support/controlplane-component/defaults_test.go#L691-L802: rename new table cases to start with When ... it should ....
  • support/controlplane-component/defaults_test.go#L921-L1072: rename new t.Run cases to start with When ... it should ....

As per coding guidelines, “Always use When ... it should ... format for describing test cases when creating unit tests.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@support/controlplane-component/defaults_test.go` around lines 691 - 802,
Rename the new table-test cases in
support/controlplane-component/defaults_test.go lines 691-802 to begin with
“When ... it should ...”, preserving their existing behavior and descriptions.
Apply the same naming format to the new t.Run cases in
support/controlplane-component/defaults_test.go lines 921-1072; update only the
test descriptions.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@support/controlplane-component/defaults.go`:
- Around line 182-189: Update the pod-template resource hashing around
computeContainerResourcesHash to include both regular containers and init
containers in separate hash fields within the resourceHashAnnotation value.
Ensure changes, including removal of init-container resources, alter the
annotation and trigger reconciliation; apply the same adjustment to the
corresponding logic near the additional referenced section.

---

Nitpick comments:
In `@support/controlplane-component/defaults_test.go`:
- Around line 691-802: Rename the new table-test cases in
support/controlplane-component/defaults_test.go lines 691-802 to begin with
“When ... it should ...”, preserving their existing behavior and descriptions.
Apply the same naming format to the new t.Run cases in
support/controlplane-component/defaults_test.go lines 921-1072; update only the
test descriptions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: eb259718-4e48-4674-816b-a0ea9a87ee30

📥 Commits

Reviewing files that changed from the base of the PR and between d255fd6 and 2e16e47.

⛔ Files ignored due to path filters (1)
  • vendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.go is excluded by !vendor/**, !**/vendor/**
📒 Files selected for processing (4)
  • api/hypershift/v1beta1/hostedcluster_types.go
  • support/controlplane-component/controlplane-component.go
  • support/controlplane-component/defaults.go
  • support/controlplane-component/defaults_test.go

Comment thread support/controlplane-component/defaults.go Outdated
@openshift-ci
openshift-ci Bot requested review from devguyio and ironcladlou July 21, 2026 16:18
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.45455% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.48%. Comparing base (97db458) to head (9ab1a15).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
support/controlplane-component/defaults.go 96.87% 1 Missing and 1 partial ⚠️
...t/controlplane-component/controlplane-component.go 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9047      +/-   ##
==========================================
+ Coverage   44.44%   44.48%   +0.03%     
==========================================
  Files         774      774              
  Lines       96977    97041      +64     
==========================================
+ Hits        43105    43166      +61     
- Misses      50897    50899       +2     
- Partials     2975     2976       +1     
Files with missing lines Coverage Δ
...t/controlplane-component/controlplane-component.go 44.63% <50.00%> (+0.31%) ⬆️
support/controlplane-component/defaults.go 71.22% <96.87%> (+3.06%) ⬆️
Flag Coverage Δ
cmd-support 38.50% <95.45%> (+0.11%) ⬆️
cpo-other 45.25% <ø> (ø)
other 32.64% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@openshift-ci openshift-ci Bot added area/api Indicates the PR includes changes for the API area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release and removed do-not-merge/needs-area labels Jul 21, 2026
@openshift-ci

openshift-ci Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@sdminonne: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@sdminonne sdminonne changed the title OCPBUGS-99272: Make container resource preservation opt-in via annotation OCPBUGS-99272: Auto-remove resource limits with user-override detection Jul 21, 2026
@openshift-ci

openshift-ci Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sdminonne
Once this PR has been reviewed and has the lgtm label, please assign csrwng for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
support/controlplane-component/defaults.go (1)

811-811: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Don't discard the json.Marshal error.

Although marshalling ResourceRequirements is unlikely to fail, silently dropping the error and hashing a possibly-empty payload would produce a misleading, stable hash. Propagate or at least log it. As per path instructions, "Never ignore error returns".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@support/controlplane-component/defaults.go` at line 811, Handle the error
returned by json.Marshal(resources) instead of discarding it. Update the
surrounding hashing flow to propagate or log the marshalling failure before
using data, ensuring a failed marshal cannot produce a misleading hash.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@support/controlplane-component/defaults_test.go`:
- Around line 844-872: The affected subtest names in the current test block,
TestMarkLimitsRemoved, and TestLimitRemovalAndPreservation do not follow the
required convention. Rename each case to use “When … it should …”, preserving
the existing scenario and expected outcome in the wording.

---

Nitpick comments:
In `@support/controlplane-component/defaults.go`:
- Line 811: Handle the error returned by json.Marshal(resources) instead of
discarding it. Update the surrounding hashing flow to propagate or log the
marshalling failure before using data, ensuring a failed marshal cannot produce
a misleading hash.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 3e1cad36-3403-4864-a24a-f82900ee305a

📥 Commits

Reviewing files that changed from the base of the PR and between 2e16e47 and 43aa88b.

📒 Files selected for processing (3)
  • support/controlplane-component/controlplane-component.go
  • support/controlplane-component/defaults.go
  • support/controlplane-component/defaults_test.go

Comment thread support/controlplane-component/defaults_test.go Outdated
@sdminonne sdminonne changed the title OCPBUGS-99272: Auto-remove resource limits with user-override detection OCPBUGS-99272: Auto-remove capi-provider resource limits with user-override detection Jul 21, 2026
@sdminonne

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@support/controlplane-component/defaults.go`:
- Line 820: Handle the error returned by json.Marshal in the resource hashing
flow instead of discarding it. Update the surrounding function to propagate the
marshaling error to its caller, preserving normal hash generation only when
marshaling succeeds.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 83547164-22df-46de-a12d-b5df58a601f4

📥 Commits

Reviewing files that changed from the base of the PR and between 0a7e9d4 and 2641a3e.

📒 Files selected for processing (3)
  • support/controlplane-component/controlplane-component.go
  • support/controlplane-component/defaults.go
  • support/controlplane-component/defaults_test.go

Comment thread support/controlplane-component/defaults.go
…erride detection

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api Indicates the PR includes changes for the API area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants