CMP-4391: Run scan-config parallel tests before serial tests with cluster reuse

[taimurhafeez]

Summary

The e2e-scan-config suite contains both parallel and serial tests. Previously, a single go test invocation ran serial tests first (Go's default behavior), which is suboptimal — parallel tests should run first to maximize concurrency, followed by the serial tests that require exclusive cluster access.

A naive two-command approach (splitting with -skip/-run) failed because each go test invocation triggers a full TestMain → SetUp() cycle, causing the second run to create a new namespace and hang waiting for ProfileBundles while the first run's environment was already torn down.

This PR solves the problem by adding two environment variables to the e2e test framework that enable cluster reuse between test runs:

CO_SKIP_SETUP: When set, SetUp() skips all cluster operations (namespace creation, CRD/RBAC deployment, operator deployment, ProfileBundle wait, MachineConfigPool setup) and only performs in-process scheme registration (addFrameworks()), which is required for the test client to work with custom resources.

CO_SKIP_TEARDOWN: When set, TearDown() returns immediately, preserving the cluster environment for the next run.

The e2e-scan-config Makefile target now runs two go test commands sharing a fixed namespace (co-e2e-scan-config via the existing TEST_OPERATOR_NAMESPACE env var):

First run: parallel tests (-skip serial), full setup, CO_SKIP_TEARDOWN=1 → cluster stays up
Second run: serial tests (-run serial), CO_SKIP_SETUP=1 → reuses cluster instantly, full teardown at the end

Changes

tests/e2e/framework/main_entry.go: Added CO_SKIP_SETUP check in SetUp() and CO_SKIP_TEARDOWN check in TearDown()
Makefile: Updated e2e-scan-config target to use two-phase execution with SCAN_CONFIG_SERIAL_TESTS and SCAN_CONFIG_NS variables

Test Results

All 17 scan-config tests pass on OCP 4.22 cluster (AWS):

Phase 1 — Parallel (11 tests, ~8 min):
TestScanStorageOutOfLimitRangeFails, TestScanWithNodeSelectorFiltersCorrectly, TestScanWithNodeSelectorNoMatches, TestScheduledSuite, TestScheduledSuitePriorityClass, TestScheduledSuiteNoStorage, TestScheduledSuiteInvalidPriorityClass, TestScheduledSuiteUpdate, TestScanSettingBindingNoStorage, TestScheduledSuiteTimeoutFail, TestScanWithCustomStorageClass

Phase 2 — Serial (6 tests, ~6.5 min):
TestScanStorageOutOfQuotaRangeFails, TestTolerations, TestSuspendScanSetting, TestSuspendScanSettingDoesNotCreateScan, TestScannerAndAPICollectorLimitsConfigurable, TestStrictNodeScanConfiguration

Total time: ~16 minutes. The second phase starts instantly (no setup wait) thanks to cluster reuse.

Notes

CO_SKIP_SETUP and CO_SKIP_TEARDOWN are opt-in environment variables — they do not affect any existing test suites
The existing TEST_OPERATOR_NAMESPACE framework variable is used to share a deterministic namespace between both runs
If the first run (parallel) fails, Make stops and the second run does not execute — same behavior as existing test suites on failure
``

[openshift-ci-robot]

@taimurhafeez: This pull request references CMP-4391 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary
The e2e-scan-config suite contains both parallel and serial tests. Previously, a single go test invocation ran serial tests first (Go's default behavior), which is suboptimal — parallel tests should run first to maximize concurrency, followed by the serial tests that require exclusive cluster access.

A naive two-command approach (splitting with -skip/-run) failed because each go test invocation triggers a full TestMain → SetUp() cycle, causing the second run to create a new namespace and hang waiting for ProfileBundles while the first run's environment was already torn down.

This PR solves the problem by adding two environment variables to the e2e test framework that enable cluster reuse between test runs:

CO_SKIP_SETUP: When set, SetUp() skips all cluster operations (namespace creation, CRD/RBAC deployment, operator deployment, ProfileBundle wait, MachineConfigPool setup) and only performs in-process scheme registration (addFrameworks()), which is required for the test client to work with custom resources.

CO_SKIP_TEARDOWN: When set, TearDown() returns immediately, preserving the cluster environment for the next run.

The e2e-scan-config Makefile target now runs two go test commands sharing a fixed namespace (co-e2e-scan-config via the existing TEST_OPERATOR_NAMESPACE env var):

First run: parallel tests (-skip serial), full setup, CO_SKIP_TEARDOWN=1 → cluster stays up
Second run: serial tests (-run serial), CO_SKIP_SETUP=1 → reuses cluster instantly, full teardown at the end
Changes
tests/e2e/framework/main_entry.go: Added CO_SKIP_SETUP check in SetUp() and CO_SKIP_TEARDOWN check in TearDown()
Makefile: Updated e2e-scan-config target to use two-phase execution with SCAN_CONFIG_SERIAL_TESTS and SCAN_CONFIG_NS variables
Test Results
All 17 scan-config tests pass on OCP 4.22 cluster (AWS):

Phase 1 — Parallel (11 tests, ~8 min):
TestScanStorageOutOfLimitRangeFails, TestScanWithNodeSelectorFiltersCorrectly, TestScanWithNodeSelectorNoMatches, TestScheduledSuite, TestScheduledSuitePriorityClass, TestScheduledSuiteNoStorage, TestScheduledSuiteInvalidPriorityClass, TestScheduledSuiteUpdate, TestScanSettingBindingNoStorage, TestScheduledSuiteTimeoutFail, TestScanWithCustomStorageClass

Phase 2 — Serial (6 tests, ~6.5 min):
TestScanStorageOutOfQuotaRangeFails, TestTolerations, TestSuspendScanSetting, TestSuspendScanSettingDoesNotCreateScan, TestScannerAndAPICollectorLimitsConfigurable, TestStrictNodeScanConfiguration

Total time: ~16 minutes. The second phase starts instantly (no setup wait) thanks to cluster reuse.

Notes
CO_SKIP_SETUP and CO_SKIP_TEARDOWN are opt-in environment variables — they do not affect any existing test suites
The existing TEST_OPERATOR_NAMESPACE framework variable is used to share a deterministic namespace between both runs
If the first run (parallel) fails, Make stops and the second run does not execute — same behavior as existing test suites on failure
``

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: taimurhafeez

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [taimurhafeez]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1286-e7dd0d3d06eae05281f407c968ee9bea02e25e26

[openshift-ci]

@taimurhafeez: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-rosa	e7dd0d3	link	true	/test e2e-rosa

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[abushkin-redhat]

Hey @taimurhafeez , everything looks good though code review did find one thing of note.

If any parallel test in phase 1 fails, the co-e2e-scan-config namespace and its cluster resources are never cleaned up:

1. Phase 1 sets CO_SKIP_TEARDOWN=1, so TearDown() is a no-op.
2. cleanUpOnError defaults to false, so TestMain doesn't call TearDown() on test failure either.
3. pipefail propagates the non-zero exit code, so make aborts before phase 2 ever runs (which would've performed the teardown).

A potential option could be to add a cleanup trap or fallback target, for example:

e2e-scan-config: e2e-set-image prep-e2e
        @TEST_OPERATOR_NAMESPACE=$(SCAN_CONFIG_NS) CO_SKIP_TEARDOWN=1 ... | tee tests/e2e-scan-config.log || \
                { TEST_OPERATOR_NAMESPACE=$(SCAN_CONFIG_NS) CO_SKIP_SETUP=1 ... TearDown only; exit 1; }
        @TEST_OPERATOR_NAMESPACE=$(SCAN_CONFIG_NS) CO_SKIP_SETUP=1 ... | tee -a tests/e2e-scan-config.log

All that being said - not sure if this is worth fixing; the CI env is going to be ephemeral anyways so a leaked namespace might not matter that much in the grand scheme of things.

[abushkin-redhat]

/lgtm

[taimurhafeez]

Hey @taimurhafeez , everything looks good though code review did find one thing of note.

If any parallel test in phase 1 fails, the co-e2e-scan-config namespace and its cluster resources are never cleaned up:

1. Phase 1 sets CO_SKIP_TEARDOWN=1, so TearDown() is a no-op.
2. cleanUpOnError defaults to false, so TestMain doesn't call TearDown() on test failure either.
3. pipefail propagates the non-zero exit code, so make aborts before phase 2 ever runs (which would've performed the teardown).

A potential option could be to add a cleanup trap or fallback target, for example:

e2e-scan-config: e2e-set-image prep-e2e
        @TEST_OPERATOR_NAMESPACE=$(SCAN_CONFIG_NS) CO_SKIP_TEARDOWN=1 ... | tee tests/e2e-scan-config.log || \
                { TEST_OPERATOR_NAMESPACE=$(SCAN_CONFIG_NS) CO_SKIP_SETUP=1 ... TearDown only; exit 1; }
        @TEST_OPERATOR_NAMESPACE=$(SCAN_CONFIG_NS) CO_SKIP_SETUP=1 ... | tee -a tests/e2e-scan-config.log

All that being said - not sure if this is worth fixing; the CI env is going to be ephemeral anyways so a leaked namespace might not matter that much in the grand scheme of things.

Thanks @abushkin-redhat! Good catch and I got your point. However, I am not sure if it's essential or not given that env is ephemeral. Maybe team might have a better insight. Will discuss.