Use metrics to display runner statuses instead of status field for EphemeralRunnerSet and AutoscalingRunnerSet#4557
Open
nikola-jokic wants to merge 1 commit into
Conversation
nikola-jokic
requested review from
Steve-Glass,
mumoshu,
rentziass and
toast-gear
as code owners
Contributor
|
Hello! Thank you for your contribution. Please review our contribution guidelines to understand the project's testing and code conventions. |
…hemeralRunnerSet and AutoscalingRunnerSet
nikola-jokic
force-pushed
the
nikola-jokic/ephemeral-runner-stats-to-metrics
branch
from
d138dfd to
5223a44
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR moves ephemeral runner lifecycle aggregation visibility from status fields on EphemeralRunnerSet / AutoscalingRunnerSet into Prometheus metrics, and introduces a shared lifecycle bucketing helper to keep the aggregation deterministic.
Changes:
- Add Prometheus gauges for additional EphemeralRunner lifecycle buckets (succeeded/outdated/deleting) and a new
SetEphemeralRunnerCountsByLifecyclesetter API. - Remove runner-count fields from
EphemeralRunnerSetStatus/AutoscalingRunnerSetStatus(types + CRDs) and update controllers/tests to stop relying on those status fields. - Introduce
AggregateEphemeralRunnerLifecycleto consistently bucket EphemeralRunners (with deletion timestamp precedence) and update controller logic to use it.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| controllers/actions.github.com/metrics/metrics.go | Adds new lifecycle gauges and replaces the old 3-bucket setter with a 6-bucket lifecycle setter. |
| controllers/actions.github.com/metrics/metrics_test.go | Adds unit tests for the new metrics setter and CommonLabels contract. |
| controllers/actions.github.com/metrics/lifecycle_metrics_integration_test.go | Adds an integration-style test that gathers metrics from the controller-runtime registry. |
| controllers/actions.github.com/ephemeralrunnerset_controller.go | Stops publishing ephemeral runner count metrics from the ERS reconciler and simplifies ERS status to Phase only. |
| controllers/actions.github.com/ephemeralrunnerset_controller_test.go | Updates tests to assert runner counts by listing EphemeralRunner objects instead of using ERS status fields. |
| controllers/actions.github.com/ephemeralrunner_lifecycle.go | Adds a shared lifecycle bucketing helper with deletion-timestamp precedence. |
| controllers/actions.github.com/ephemeralrunner_lifecycle_test.go | Adds thorough unit tests for lifecycle bucketing precedence and totals. |
| controllers/actions.github.com/ephemeralrunner_controller.go | Emits lifecycle metrics on runner lifecycle/status changes and adds a PublishMetrics gate to the reconciler. |
| controllers/actions.github.com/autoscalingrunnerset_controller.go | Replaces “active runners” check based on ERS status fields with a list+aggregate of EphemeralRunners. |
| controllers/actions.github.com/autoscalingrunnerset_controller_test.go | Updates tests to stop asserting autoscaling status runner-count fields. |
| config/crd/bases/actions.github.com_ephemeralrunnersets.yaml | Removes ERS status count fields/printcolumns from the CRD schema. |
| config/crd/bases/actions.github.com_autoscalingrunnersets.yaml | Removes ARS status count fields/printcolumns from the CRD schema. |
| charts/gha-runner-scale-set-controller/crds/actions.github.com_ephemeralrunnersets.yaml | Mirrors the ERS CRD schema changes in the Helm chart CRDs. |
| charts/gha-runner-scale-set-controller/crds/actions.github.com_autoscalingrunnersets.yaml | Mirrors the ARS CRD schema changes in the Helm chart CRDs. |
| charts/gha-runner-scale-set-controller-experimental/crds/actions.github.com_ephemeralrunnersets.yaml | Mirrors the ERS CRD schema changes in the experimental chart CRDs. |
| charts/gha-runner-scale-set-controller-experimental/crds/actions.github.com_autoscalingrunnersets.yaml | Mirrors the ARS CRD schema changes in the experimental chart CRDs. |
| apis/actions.github.com/v1alpha1/ephemeralrunnerset_types.go | Removes ERS status count fields and associated kubebuilder printcolumns. |
| apis/actions.github.com/v1alpha1/autoscalingrunnerset_types.go | Removes ARS status count fields and associated kubebuilder printcolumns. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
47
to
53
| // EphemeralRunnerReconciler reconciles a EphemeralRunner object | ||
| type EphemeralRunnerReconciler struct { | ||
| client.Client | ||
| Log logr.Logger | ||
| Scheme *runtime.Scheme | ||
| Log logr.Logger | ||
| Scheme *runtime.Scheme | ||
| PublishMetrics bool | ||
| ResourceBuilder |
Comment on lines
+904
to
+907
| // emitLifecycleMetrics recomputes and emits lifecycle metrics for all EphemeralRunners | ||
| // owned by the same EphemeralRunnerSet as the given runner. This ensures metrics reflect | ||
| // the current lifecycle state of all sibling runners under the same AutoscalingRunnerSet. | ||
| // |
Comment on lines
+932
to
+933
| // Aggregate lifecycle counts using the helper from Task 3 | ||
| buckets := AggregateEphemeralRunnerLifecycle(runnerList.Items) |
Comment on lines
+107
to
+116
| // SetEphemeralRunnerCountsByLifecycle sets ephemeral runner counts across all six lifecycle states | ||
| // using deterministic bucket assignment with explicit precedence: | ||
| // | ||
| // Lifecycle Precedence Contract: | ||
| // 1. deleting: if runner has DeletionTimestamp set | ||
| // 2. explicit phase buckets: if runner.Status.Phase is one of (Running/Succeeded/Failed/Outdated) | ||
| // 3. pending (fallback): for empty/unset/other phase values | ||
| // | ||
| // This ensures each runner maps to exactly one metric bucket (no double-counting). | ||
| func SetEphemeralRunnerCountsByLifecycle(commonLabels CommonLabels, pending, running, succeeded, failed, outdated, deleting int) { |
Comment on lines
+61
to
+66
| succeededEphemeralRunners = prometheus.NewGaugeVec( | ||
| prometheus.GaugeOpts{ | ||
| Subsystem: githubScaleSetControllerSubsystem, | ||
| Name: "succeeded_ephemeral_runners", | ||
| Help: "Number of ephemeral runners in a succeeded state.", | ||
| }, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There is no need for updating the status at points to display information about ephemeral runner state aggregations. Metrics are way more useful place to do it. Therefore, we should move this information to a metric.