ci(minicloud): run minicloud on a local KVM Jenkins agent

[fruch]

What

The local-agent half of minicloud CI, split out of #15618 per review — that PR now runs every
minicloud job on an sct-runner, and this one adds the alternative: a long-lived KVM-capable
Jenkins agent with no runner at all.

Stacked on #15618 (base is its branch); only the two commits below are in this diff.

1. jenkins_label — pin any build to a named agent

Backend-agnostic and independent of minicloud: declaring the parameter is the whole integration,
because getJenkinsLabels already reads the global params binding and it is empty on build #1,
where the label is first evaluated. The pinned return resolves the effective backend region first
(datacenter/location/oci_region for GCE/Azure/OCI, and random from the per-provider
supported list) — the pipelines hand that value straight to createSctRunner and the
collect/clean stages, so a label alone is never enough.

2. local_agent — the local topology, per job or per build

Selectable two ways, so moving an existing job to a lab machine needs no new jenkinsfile:

• jenkinsfile: local_agent: true alongside minicloud: true, for a job that always runs there.
• build parameter: tick local_agent on any minicloud job and the KVM label is resolved for
  that run. Read off params from build sdcm.tester: Introduce the clean_aws_resources decorator #2 onward (build AWS use spot instances #1 only loads parameters and these
  pipelines abort it), falling back to the jenkinsfile default before that. Safe to read there
  unlike params.minicloud, since getJenkinsLabels only ever merges a minicloud key into the
  global binding.

jenkins_label remains the manual override for pinning a build to a named agent on any backend.

Runner stages are skipped and every stage takes the builder-local branch, selected as everywhere
else by the absence of ./sct_runner_ip. The cloud runner stays the default, and remains the only
option for anyone without a lab machine.

Three helpers exist only because the agent outlives its build:

helper	why
minicloudPreflight	one pass, every problem at once (KVM writability, docker daemon, minicloud0, port 5000, disk, $USER/$HOME) — before Argus registration and the hydra pull, so a misconfigured agent costs seconds, not 20 minutes
minicloudReclaim	sweeps at build start, so the previous failure's logs survive for post-mortem; never touches ~/.cache/minicloud/amis (tens of GiB, tens of minutes to rebuild — the whole economic case for a static agent). Also the only thing clearing a stale ./sct_runner_ip from a persistent workspace
stopMinicloud	local teardown, last, after log collection — docker rm -f kills every guest with the container. A no-op in runner topology, where the container dies with the instance

Also loads the KVM module at runner creation for the nested-virt case: the runner image does not
auto-load it and hydra cannot compensate from inside a container without the host's
/lib/modules. Guarded on minicloud activation so no existing job's runner runs a new command.
Kept here per review — QATOOLS-374 tracks baking it into the sct-runner image instead, after
which this can go.

Not verified end to end

No Jenkins node serves minicloud-kvm-builders-v1 yet, so this path has never run; the
prerequisites are documented in docs/minicloud.md. The seam itself is provable today by pinning
any job with jenkins_label. Everything in #15618 is verified — AWS and GCE artifact jobs
pass on the runner topology.

Open question from review

@fruch asked whether the preflight logic should be a hydra command reusing
MinicloudManager.preflight_check() instead of bash. Partly yes — /dev/kvm, docker and memory
are duplicated. But the value of this stage is failing in ~10 s before the hydra image pull, and
a hydra command would need that image first; the agent-specific checks (minicloud0, port 5000,
group membership, disk) have no Python equivalent either. Happy to go either way — flagging the
trade-off rather than picking silently.

🤖 Generated with Claude Code

---

Note

Bottom of a stack that merges one PR at a time, in order: #15617 (core, targets master, self-contained) → #15618 (pipelines, sct-runner topology) → this. Each is rebased onto master and reviewed on its own after the one below it lands; none of them should be merged together. jenkins/precommit fails here only because of the stacking — Jenkins derives baseHash from a merge ref it cannot reach for a stacked PR — and clears once the base is master.

[fruch]

Carrying over a security finding from the review of #15618, since jenkins_label moved here:

jenkins_label is a free-form build parameter returned directly as the Pipeline node label. Jenkins treats this value as a label expression, so a build user can select arbitrary agents (including sensitive labels or expressions using ||) for jobs that bind QA credentials.

Worth addressing before this merges — an allowlist of approved labels plus rejecting expression syntax (||, &&, parentheses), or making the pin jenkinsfile-controlled rather than a build parameter. The same class of issue on the minicloud image reference is already fixed in #15618 with a repository allowlist validated before export.

[scylladb-promoter]

✅ Test Summary: PASSED

✅ Precommit: PASSED

Total	Passed	Failed	Skipped
15	15	0	0

✅ Tests: PASSED

Total	Passed	Failed	Errors	Skipped
4782	4751	0	0	31

Full build log

[Copilot]

Pull request overview

Adds local KVM Jenkins-agent support for minicloud while retaining cloud runner topology.

Changes:

• Adds agent selection and local topology controls.
• Adds preflight, reclaim, and teardown helpers.
• Documents local-agent requirements and loads KVM modules on runners.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
vars/stopMinicloud.groovy	Archives logs and removes minicloud.
vars/rollingUpgradePipeline.groovy	Adds local-agent upgrade topology.
vars/minicloudReclaim.groovy	Cleans persistent-agent state.
vars/minicloudPreflight.groovy	Validates local-agent prerequisites.
vars/longevityPipeline.groovy	Adds local-agent longevity topology.
vars/getJenkinsLabels.groovy	Supports pinned and KVM labels.
vars/artifactsPipeline.groovy	Adds selectable minicloud topology.
sdcm/utils/lint/env_builder.py	Excludes orchestration parameter from SCT environment.
sct.py	Loads KVM modules on minicloud runners.
docs/sct-pipelines.md	Documents local pipeline execution.
docs/minicloud.md	Documents topology and prerequisites.

Suppressed comments (1)

docs/sct-pipelines.md:63

• The artifacts pipeline no longer mandates local topology: local_agent defaults to false and useRunner then provisions the nested-virtualization runner. This sentence would send users to expect the opposite behavior.

For minicloud specifically - which needs a KVM-capable agent and mandates the local topology
on artifacts - see [minicloud](./minicloud.md). For the cloud sct-runner mechanics, see
[sct-runners](./sct-runners.md).

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (12)

vars/longevityPipeline.groovy:25

• localAgent is honored even when minicloudEnabled is false, although the new parameter says it is ignored then. Ticking it on an ordinary longevity build therefore skips runner creation and cleanup and silently runs on the normal regional builder. Either gate the effective value with minicloudEnabled, or explicitly require a suitable jenkins_label and update the parameter contract.

    def localAgent = (params.local_agent != null ? params.local_agent.toString().toBoolean()
                                                 : pipelineParams.get('local_agent', false))

vars/getJenkinsLabels.groovy:130

• This also assigns AWS eu-west-1 to local minicloud runs whose GCE/Azure/OCI region parameter is empty. Since builder.region is forwarded to provisioning and cleanup, resolve the fallback from the provider-specific defaults already represented by the generic labels above.

        return [ "label": minicloudLabel, "region": effectiveRegion ?: 'eu-west-1' ]

vars/rollingUpgradePipeline.groovy:22

• localAgent is honored even when minicloudEnabled is false, despite the parameter being documented as ignored. On an ordinary rolling-upgrade build this suppresses runner creation and both cleanup paths, causing the test to execute on the regional builder. Gate local mode on minicloud, or validate that a deliberate jenkins_label was supplied and document that generic mode.

    def localAgent = (params.local_agent != null ? params.local_agent.toString().toBoolean()
                                                 : pipelineParams.get('local_agent', false))

docs/sct-pipelines.md:50

• local_agent is now also a build parameter in all three pipelines, so calling it only a jenkinsfile literal and omitting artifacts is outdated.

- **`local_agent: true` (jenkinsfile literal, longevity / rolling-upgrade)** - skip the

vars/longevityPipeline.groovy:409

• This preflight runs only after Checkout/docker login, Argus registration, timeout discovery, and the potentially 240-minute BYO stage. That contradicts the PR/docs guarantee that a bad agent fails before Argus and expensive work. Move reclaim/preflight ahead of Create Argus Test Run and BYO for local minicloud builds.

            stage('Minicloud Agent Preflight') {
                when { expression { minicloudEnabled && localAgent } }

vars/rollingUpgradePipeline.groovy:337

• In each rolling-upgrade branch, Argus is registered before this preflight, and the top-level supported-version/BYO stages have already run. Thus a misconfigured local agent does not fail before Argus and expensive work as promised. Run a local-agent preflight before those stages (and before per-version Argus registration).

                                                } else if (minicloudEnabled) {
                                                    stage("Minicloud Agent Preflight for ${base_version}") {

docs/sct-pipelines.md:59

• This rationale is no longer true: the implementation explicitly reads params.local_agent from build #2 onward. Document the build-1 fallback instead of saying the option cannot be a job parameter.

Why a jenkinsfile literal and not a job parameter: the agent label is decided at
pipeline-definition time, before parameters exist on build #1 - the same reason
`functional_test` works the way it does.

docs/minicloud.md:238

• The preflight does not verify numExecutors=1, exclusive mode, or any of the listed egress destinations, so saying it verifies all prerequisites can leave an agent incorrectly configured. Distinguish the host-local checks from the Jenkins and network settings that remain manual.

egress to docker.io, ghcr.io, github.com, amazonaws.com, argus.scylladb.com,
downloads.scylladb.com. The *Minicloud Preflight* stage verifies all of this in one pass.

vars/getJenkinsLabels.groovy:117

• When a pinned GCE/Azure/OCI build omits its backend-specific region, this fallback returns AWS eu-west-1. The pipeline then passes that invalid region to runner creation and cleanup. Use the existing provider defaults (us-east1, eastus, us-phoenix-1) rather than an AWS-only fallback.

This issue also appears on line 130 of the same file.

        println("Pinned to Jenkins agent label: " + pinnedLabel + ", region: " + (effectiveRegion ?: 'eu-west-1'))
        return [ "label": pinnedLabel, "region": effectiveRegion ?: 'eu-west-1' ]

vars/artifactsPipeline.groovy:24

• This makes the cloud runner the default for minicloud artifact jobs, but vars/startMinicloud.groovy:24-29 and several minicloud jenkinsfile headers still document artifacts as local-agent-by-default and say local_agent: false opts into a runner. Those instructions now select the opposite topology and should be updated with this behavior change.

    def useRunner = minicloudEnabled && !localAgent

docs/sct-pipelines.md:63

• Artifacts no longer mandate local topology: useRunner = minicloudEnabled && !localAgent makes the cloud runner the default. This sentence currently tells readers the opposite.

For minicloud specifically - which needs a KVM-capable agent and mandates the local topology
on artifacts - see [minicloud](./minicloud.md). For the cloud sct-runner mechanics, see
[sct-runners](./sct-runners.md).

sct.py:2943

• A timeout or transport error from this new modprobe call is caught by the existing outer handler, which reports “Unable to SSH” even though SSH verification already succeeded. Use an initialization-level error message (or catch this operation separately) so nested-virtualization failures are diagnosable.

            remoter.sudo(
                "modprobe kvm_intel || modprobe kvm_amd || true", timeout=60, ignore_status=True, verbose=False

[fruch]

Restacked onto #15618's new head (dea951045d). Still 2 commits / 10 files of its own.

Merge order note: this is the bottom of a stack that merges one PR at a time — #15617 (core, targets master, self-contained) → #15618 (pipelines, sct-runner topology) → this. Each is rebased onto master and reviewed on its own after the one below it lands; none of them get merged together. jenkins/precommit fails here purely because of the stacking and clears once the base is master.

Sequence and rationale: docs/plans/minicloud-stack-merge-runbook.md (on #15617).

[scylladbbot]

@fruch new branch manager-3.12 was added, please add backport label if needed