CMP-4373: Fix stale compliance_state metric and suite result

[Vincent056]

What

The ComplianceSuite controller could report a stale compliance result —
oc get compliancesuite / the compliance_operator_compliance_state gauge /
ScanSettingBinding events showing NON-COMPLIANT while the underlying
ComplianceScan was COMPLIANT — until the operator pod was restarted. This is CMP-4373.

Root cause

updateScanStatus refreshed the suite status (and therefore the metric and the
events derived from it) only on a scan phase transition:

if scanStatusWrap.Phase == scan.Status.Phase {
    return nil
}

A rescan returns a scan to the same terminal phase (DONE) it started from, so
when only the result changed (e.g. NON-COMPLIANT → COMPLIANT after
remediation) the phase-only check skipped the update and the suite result went stale.

Two related gauge-lifecycle gaps made it worse: compliance_state lives only in
the operator's in-memory registry and was written exclusively on that transition
path and never cleaned up — so a live suite could be missing from metrics after
a restart, and a deleted suite left a stale series firing alerts indefinitely.

Fix

• Compare the scan result as well as the phase in updateScanStatus.
• Re-assert compliance_state from the authoritative suite.Status.Result on
  every reconcile of a Done suite (survives operator restart).
• Delete the compliance_state series for a suite in suiteDeleteHandler.

Tests

• New unit regression test: a scan whose result flips while its phase stays DONE
  must update the suite result. Fails on master, passes with this change.
• Verified end-to-end on an OCP 4.22 cluster: reproduced all three behaviours on
  the released 1.9.1 image (stale value, absent-after-restart, ghost-on-delete) and
  confirmed each is resolved by this build.

Jira

CMP-4373

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@Vincent056: This pull request references CMP-4373 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What

The ComplianceSuite controller could report a stale compliance result —
oc get compliancesuite / the compliance_operator_compliance_state gauge /
ScanSettingBinding events showing NON-COMPLIANT while the underlying
ComplianceScan was COMPLIANT — until the operator pod was restarted. This is CMP-4373.

Root cause

updateScanStatus refreshed the suite status (and therefore the metric and the
events derived from it) only on a scan phase transition:

if scanStatusWrap.Phase == scan.Status.Phase {
   return nil
}

A rescan returns a scan to the same terminal phase (DONE) it started from, so
when only the result changed (e.g. NON-COMPLIANT → COMPLIANT after
remediation) the phase-only check skipped the update and the suite result went stale.

Two related gauge-lifecycle gaps made it worse: compliance_state lives only in
the operator's in-memory registry and was written exclusively on that transition
path and never cleaned up — so a live suite could be missing from metrics after
a restart, and a deleted suite left a stale series firing alerts indefinitely.

Fix

• Compare the scan result as well as the phase in updateScanStatus.
• Re-assert compliance_state from the authoritative suite.Status.Result on
  every reconcile of a Done suite (survives operator restart).
• Delete the compliance_state series for a suite in suiteDeleteHandler.

Tests

• New unit regression test: a scan whose result flips while its phase stays DONE
  must update the suite result. Fails on master, passes with this change.
• Verified end-to-end on an OCP 4.22 cluster: reproduced all three behaviours on
  the released 1.9.1 image (stale value, absent-after-restart, ghost-on-delete) and
  confirmed each is resolved by this build.

Jira

CMP-4373

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Vincent056

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [Vincent056]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1263-feb92d872c724476b070d3ef27413df1d33fc3cc

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1263-6a572494276caecc0c07b3579f0eaa03533f1185

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1263-f00b444a1bb0d826eee95b7256b2cc783e8e174e

[taimurhafeez]

/test e2e-aws-parallel

[openshift-ci]

@Vincent056: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-parallel	f00b444	link	true	/test e2e-aws-parallel

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.