Feature OCPSTRAT-3156: Configurable log levels for hosted control plane components#2038
Feature OCPSTRAT-3156: Configurable log levels for hosted control plane components#2038rutvik23 wants to merge 1 commit into
Conversation
Introduces an enhancement proposal for structured, per-component log level configuration (Normal/Debug/Trace/TraceAll) for hosted control plane components managed by CPO via the HostedCluster CR. Covers kube-apiserver (Phase 1), 7 remaining core components (Phase 2), and documents holistic scope for all CPO-managed components (Phase 3). Replaces the existing ad-hoc kube-apiserver-verbosity-level annotation with a validated, structured API using ComponentLogLevelSpec pointer fields in OperatorConfiguration. Tracking: https://issues.redhat.com/browse/OCPSTRAT-3156 RFE: https://issues.redhat.com/browse/RFE-7777
|
Skipping CI for Draft Pull Request. |
openshift-ci-robot
added
the
jira/valid-reference
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
@rutvik23: This pull request references OCPSTRAT-3156 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the feature to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
| as structured `*ComponentLogLevelSpec` pointers in the existing `OperatorConfiguration` | ||
| type. The design is intentionally generic to extend to all remaining CPO-managed components | ||
| in future phases. This feature ships ungated (no feature gate) as a low-risk operational | ||
| knob that extends an already-GA framework. |
There was a problem hiding this comment.
The existing ClusterVersionOperator field in OperatorConfiguration IS gated behind +openshift:enable:FeatureGate=ClusterVersionOperatorConfiguration. The API review guide states: "New fields in existing APIs should always be gated." Skipping the gate here is inconsistent with the precedent set by the only other log-level field in this same struct. What is the justification for treating these 8 new fields differently from the CVO field that lives right next to them?
| // +kubebuilder:default=Normal | ||
| // +kubebuilder:validation:Enum="";Normal;Debug;Trace;TraceAll | ||
| LogLevel LogLevel `json:"logLevel,omitempty"` | ||
| } |
There was a problem hiding this comment.
Several issues with this type:
-
Static default on a configuration API.
+kubebuilder:default=Normalpersists the default at admission time. Per OpenShift API conventions, configuration APIs should default in the controller, not the schema, so the default can evolve over time. IfNormalis intended to be permanent, document that explicitly. Otherwise, remove the marker and default in the CPO, with Godoc: "When omitted, this means the user has no opinion and the platform defaults to Normal, which is subject to change over time." -
Empty string in enum is ambiguous. The enum includes
""alongside+kubebuilder:default=Normal. If a user submitskubeAPIServer: {}, the field defaults to"Normal". If they explicitly submitlogLevel: "", does that differ from omitting it? The OpenShift convention is: "Only allow the zero value when it is semantically important and means something different to being omitted." What is the semantic meaning of""here? -
Single-field wrapper struct. If
ComponentLogLevelSpecwill only ever containlogLevel, consider using*LogLeveldirectly on theOperatorConfigurationfields (e.g.,KubeAPIServerLogLevel *LogLevel). This is simpler for users (kubeAPIServerLogLevel: DebugvskubeAPIServer: {logLevel: Debug}) and avoids the extensibility trap described below.
| // oAuthServer configures the log verbosity of the oauth-server component. | ||
| // +optional | ||
| OAuthServer *ComponentLogLevelSpec `json:"oAuthServer,omitempty"` | ||
| } |
There was a problem hiding this comment.
These field names (kubeAPIServer, etcd, openShiftAPIServer, etc.) are generic, implying they are the full configuration entry point for that component. But the type is *ComponentLogLevelSpec, which only carries a log level.
If any of these 8 components ever needs additional per-component configuration (resource overrides, feature flags, etc.), you cannot change the type from *ComponentLogLevelSpec to a dedicated struct without breaking structured clients (serialization changes are forbidden post-merge per API review policy).
You already acknowledge this problem for CVO/CNO/Ingress in the Phase 3 note (line 444-454), where those components get LogLevel embedded in their existing dedicated typed structs. That is the correct pattern. Consider one of:
- Use dedicated per-component types from the start (
KubeAPIServerConfig,EtcdConfig, etc.) each with alogLevelfield. More boilerplate now, but future-proof. - Or rename the fields to be specific:
kubeAPIServerLogLevel *LogLevel,etcdLogLevel *LogLevel, etc. Simpler and leaves namespace for futurekubeAPIServer *KubeAPIServerConfigfields.
| type OperatorConfiguration struct { | ||
| // ...existing ClusterVersionOperator, ClusterNetworkOperator, IngressOperator fields... | ||
|
|
||
| // kubeAPIServer configures the log verbosity of the kube-apiserver component. |
There was a problem hiding this comment.
The Godoc should describe user-observable behavior per OpenShift API conventions. Currently it says "configures the log verbosity," which is implementation-oriented. Consider adding:
- What happens when the field is set (rolling restart of the component, increased log volume)
- What happens when the field is omitted: "When omitted, this means the user has no opinion and the platform defaults to Normal verbosity."
The existing clusterVersionOperator field's Godoc in OperatorConfiguration follows this pattern with behavioral documentation on operatorLogLevel.
|
|
||
| For managed services (ROSA HCP, ARO HCP), service-level guardrails may be needed to | ||
| restrict `TraceAll` log level, as glog level 8 can dump sensitive data including request | ||
| bodies and secrets in component logs. |
There was a problem hiding this comment.
"May be needed" is too weak for a GA feature. TraceAll at glog level 8 dumps request bodies including secrets. On ROSA HCP, those logs flow to CloudWatch, which is customer-visible. This is a concrete data-leak vector, not a hypothetical.
Before GA, the enhancement should commit to one of:
- Block
TraceAllon managed service clusters (admission webhook or CEL) - Allow it but ensure logs containing secrets are not forwarded to customer-visible sinks
- Accept the risk with explicit documented warnings
Deferring the decision to "be decided with PM" (Open Question 1) is not acceptable for a feature shipping GA without a feature gate.
| **Risk:** `TraceAll` on managed services (ROSA HCP, ARO HCP) could expose secrets in logs | ||
| flowing to CloudWatch or Azure Monitor (customer-visible). | ||
| **Mitigation:** Add admission policy or CEL rule blocking `TraceAll` on managed service | ||
| clusters. Access model (customer self-service vs SRE/CEE only) to be decided with PM. |
There was a problem hiding this comment.
This mitigation is stated in imperative ("Add admission policy") but is not listed in the GA Criteria (lines 583-598) or the Phase 1/Phase 2 deliverables. Is this a commitment or a suggestion? If it's required for GA, add it to the GA Criteria. If it's deferred, say so explicitly and explain why shipping GA without it is acceptable given the security implications.
| **Risk:** Rolling restarts during log level changes could cause brief API unavailability if | ||
| multiple components are changed simultaneously. | ||
| **Mitigation:** The CPO should apply changes sequentially across components to avoid | ||
| simultaneous restarts. |
There was a problem hiding this comment.
"Should" is RFC 2119 advisory, not mandatory. If a user patches all 8 components in one oc patch, will the CPO actually serialize the rollouts? Or will it update all deployments in a single reconciliation pass? This matters for the HA guarantee. If sequential application is required for safety, state it as "must" and describe the implementation mechanism (e.g., reconcile one component per pass, proceed only after rollout completes).
|
|
||
| The existing `hypershift.openshift.io/kube-apiserver-verbosity-level` annotation is | ||
| deprecated on merge of this feature and will be removed in N+2 (two minor releases after | ||
| GA). |
There was a problem hiding this comment.
"N+2" is ambiguous since this EP doesn't specify what "N" is and the Jira target version is not set (the CI bot flagged this). Use concrete OCP versions (e.g., "deprecated in 4.19, removed in 4.21") so readers don't have to cross-reference the target release.
| with elevated verbosity. Administrators and SREs can alert on this to track clusters with | ||
| non-standard log volume. | ||
| - `HostedCluster` condition `NonDefaultLogLevel=False` — all components running at default | ||
| verbosity. |
There was a problem hiding this comment.
The condition is described but missing key details:
-
What
Reasonvalues are expected? Per Kubernetes API conventions,Reasonis required and should be a CamelCase programmatic identifier. Define the expected values (e.g.,ElevatedVerbosity,AllDefault). -
The condition message should enumerate which components have non-default levels.
NonDefaultLogLevel=Truewithout component detail forces SREs to inspect the full spec to find which component is elevated. Include something like "kube-apiserver: Debug, etcd: Trace" in the message.
| > via a shell one-liner (`/bin/sh -c "... /usr/bin/etcd"`). Injecting a CLI flag means | ||
| > string-manipulating the shell command — fragile and breaks if the command format changes. | ||
| > `ETCD_LOG_LEVEL` matches all 15 existing etcd config values and uses the existing | ||
| > `util.UpsertEnvVar()` utility (from `support/util/containers.go`). etcd internally maps |
There was a problem hiding this comment.
Minor: the file path is incorrect. UpsertEnvVar lives in support/podspec/containers.go, not support/util/containers.go. There is no containers.go in support/util/.
3 participants
Why
HyperShift administrators and support engineers cannot adjust log
verbosity for hosted control plane components during troubleshooting.
Standard OCP has this via operatorv1.LogLevel, but HyperShift has no
equivalent except an ad-hoc, unvalidated annotation for kube-apiserver.
This was reported as RFE-7777 and accepted as crucial for minimal
troubleshooting.
What
Enhancement proposal introducing structured per-component log level
configuration for CPO-managed components via the HostedCluster CR:
ComponentLogLevelSpectype withNormal/Debug/Trace/TraceAllenum (matching operatorv1.LogLevel)
OperatorConfigurationfor 8 corecontrol plane components
(Phase 2), all remaining CPO-managed components documented
holistically (Phase 3)
annotation
References