Skip to content

CNTRLPLANE-3774: [release-4.22] fix: control-plane-operator: improve opaque conditions#9058

Open
csrwng wants to merge 7 commits into
openshift:release-4.22from
csrwng:backport-8804-release-4.22
Open

CNTRLPLANE-3774: [release-4.22] fix: control-plane-operator: improve opaque conditions#9058
csrwng wants to merge 7 commits into
openshift:release-4.22from
csrwng:backport-8804-release-4.22

Conversation

@csrwng

@csrwng csrwng commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Backport of #8804 to release-4.22.

Summary

  • Improved KubeAPI health checks to use verbose /healthz?verbose output, surfacing which specific health checks are failing and adding endpoint context on request failures
  • Improved load balancer and endpoint health diagnostics, distinguishing "not yet provisioned" from "missing ingress hostname/IP" with event details
  • Expanded Route-based readiness errors with richer ingress/admission condition details when required routing attributes are absent
  • More reliable Route status reconciliation by preventing overwrites when router hostnames aren't known
  • Added hypershift.openshift.io/route-status-writer annotation to attribute route status writes to either "shared-ingress" or "hcp-router"

Backport Adaptations

  • Replaced netutil.RouteStatusWriterAnnotation with util.RouteStatusWriterAnnotation since support/netutil doesn't exist on release-4.22
  • Inlined k8sutil.CollectLBMessageIfNotProvisioned since support/k8sutil doesn't exist on release-4.22
  • Updated healthCheckKASEndpoint callers to pass ctx parameter (function signature change was part of the PR but callers needed manual fixup due to cherry-pick context differences)

stevekuznetsov and others added 7 commits July 22, 2026 16:37
When we have the context on *why* something is not progressing or
presenting the way we expected it to be, we should say so. Users will be
thrilled to not need to learn about which code flows might have been
triggered and think through the implicit reasoning behind why a
condition is failing.

Signed-off-by: Steve Kuznetsov <stekuznetsov@microsoft.com>
…ostname

When the shared ingress LB service has no hostname or IP, the controller
was writing an empty RouterCanonicalHostname to all HCP-labeled routes.
This overwrites any previously-valid value and causes the CPO health
check to report "route not admitted", flapping the HostedCluster
Available condition every reconcile cycle.

Skip route status reconciliation entirely when the canonical hostname
is empty, preserving the last-known-good route status.

Signed-off-by: Steve Kuznetsov <stekuznetsov@microsoft.com>
Commit-Message-Assisted-by: Claude (via Claude Code)
…n logs

Differentiate the two sub-cases when skipping route status
reconciliation: LB has no ingress entries at all (not yet provisioned,
includes creation timestamp) vs LB has ingress entries with no
hostname or IP. This lets operators diagnose the root cause from shared
ingress controller logs alone.

Signed-off-by: Steve Kuznetsov <stekuznetsov@microsoft.com>
Commit-Message-Assisted-by: Claude (via Claude Code)
Both the shared ingress controller and the CPO HCP router write route
Status.Ingress with RouterName "router", making their writes
indistinguishable when diagnosing route status flapping.

Add a hypershift.openshift.io/route-status-writer annotation set to
"shared-ingress" or "hcp-router" by the respective controllers, and
include its value in the CPO health check error message.

Signed-off-by: Steve Kuznetsov <stekuznetsov@microsoft.com>
Commit-Message-Assisted-by: Claude (via Claude Code)
The "route has no ingress status" case previously gave no indication of
whether the route was never admitted or had its admission cleared. By
including the route-status-writer annotation value, operators can tell
from the HostedCluster condition message alone which controller last
touched the route, without needing to inspect the route object.

Signed-off-by: Steve Kuznetsov <stekuznetsov@microsoft.com>
Commit-Message-Assisted-by: Claude (via Claude Code)
Signed-off-by: Steve Kuznetsov <stekuznetsov@microsoft.com>
Signed-off-by: Steve Kuznetsov <stekuznetsov@microsoft.com>
Replace netutil.RouteStatusWriterAnnotation with util.RouteStatusWriterAnnotation
since the support/netutil package does not exist on the release-4.22 branch.
Inline k8sutil.CollectLBMessageIfNotProvisioned since the support/k8sutil package
does not exist on the release-4.22 branch.

Signed-off-by: Cesar Wong <cewong@redhat.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 22, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 22, 2026

Copy link
Copy Markdown

@csrwng: This pull request references CNTRLPLANE-3774 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target either version "4.22." or "openshift-4.22.", but it targets "openshift-5.0" instead.

Details

In response to this:

Backport of #8804 to release-4.22.

Summary

  • Improved KubeAPI health checks to use verbose /healthz?verbose output, surfacing which specific health checks are failing and adding endpoint context on request failures
  • Improved load balancer and endpoint health diagnostics, distinguishing "not yet provisioned" from "missing ingress hostname/IP" with event details
  • Expanded Route-based readiness errors with richer ingress/admission condition details when required routing attributes are absent
  • More reliable Route status reconciliation by preventing overwrites when router hostnames aren't known
  • Added hypershift.openshift.io/route-status-writer annotation to attribute route status writes to either "shared-ingress" or "hcp-router"

Backport Adaptations

  • Replaced netutil.RouteStatusWriterAnnotation with util.RouteStatusWriterAnnotation since support/netutil doesn't exist on release-4.22
  • Inlined k8sutil.CollectLBMessageIfNotProvisioned since support/k8sutil doesn't exist on release-4.22
  • Updated healthCheckKASEndpoint callers to pass ctx parameter (function signature change was part of the PR but callers needed manual fixup due to cherry-pick context differences)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 71f8e855-5bbf-4b14-994e-4cf23cb1af5d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from bryan-cox and devguyio July 22, 2026 14:46
@openshift-ci openshift-ci Bot added the area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release label Jul 22, 2026
@openshift-ci

openshift-ci Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed do-not-merge/needs-area labels Jul 22, 2026
@openshift-ci

openshift-ci Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

@csrwng: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants