ci(minicloud): run minicloud on a local KVM Jenkins agent

docs/minicloud.md

@@ -253,8 +253,44 @@ longevity/rolling-upgrade test-cases set it themselves, and the artifacts jobs l
 'Clean SCT Runners' - which takes the container and every guest with it, so it runs after log
 collection.
 
-Running minicloud on a long-lived KVM Jenkins agent instead is a follow-up; it needs agent
-validation, workspace reclaim and a local teardown path that this PR deliberately leaves out.
+### The local-agent topology
+
+`local_agent: true` in the jenkinsfile runs everything on a KVM-capable Jenkins agent instead
+(label `minicloud-kvm-builders-v1`), with no sct-runner at all: the runner stages are skipped and
+every stage takes the builder-local branch, selected as everywhere else by the absence of
+`./sct_runner_ip`. The build gains *Minicloud Reclaim → Minicloud Preflight* ahead of
+*Start Minicloud*, and *Stop Minicloud* last - after log collection, because `docker rm -f` kills
+every guest with the container.
+
+Two things only a long-lived agent needs, both by design:
+
+- **Reclaim at build start, not at the end.** The previous failure's logs stay on the box for
+  post-mortem, which is one of the few genuine advantages of a static agent. The AMI cache
+  (`~/.cache/minicloud/amis`) is never swept - it is tens of GiB and tens of minutes to rebuild,
+  and it is the entire economic case for the agent.
+- **Preflight in one pass.** A misconfigured agent reports every problem at once rather than one
+  per build, and it runs before Argus registration and the hydra pull so a bad agent costs
+  seconds instead of twenty minutes.
+
+`local_agent` is a **job parameter** as well as a jenkinsfile knob, so an existing minicloud job
+can be moved to a lab agent for a single run without a new jenkinsfile: tick it and the KVM label
+is resolved for you. It is read off `params` from build #2 onward - build #1 only loads parameters
+and these pipelines abort it - falling back to the jenkinsfile default before that. `jenkins_label`
+stays the manual override for pinning any build to an approved agent label on any backend,
+independent of minicloud; see [sct-pipelines](./sct-pipelines.md).
+
+### Agent prerequisites
+
+A lab agent serving `minicloud-kvm-builders-v1` needs: the agent user in `kvm` and `docker`
+groups (restart the agent process after `usermod`, reconnecting is not enough); `minicloud0`
+pre-created by a boot-time unit (preferred - no sudo needed at run time) or passwordless sudo;
+`USER`/`HOME` set and `$HOME` writable with >=80 GiB free; `numExecutors=1` + exclusive mode,
+which is what serialises the host singletons (port 5000, the container name, `minicloud0`);
+egress to docker.io, ghcr.io, github.com, amazonaws.com, argus.scylladb.com,
+downloads.scylladb.com. The *Minicloud Preflight* stage verifies what a shell on the agent can
+see - /dev/kvm writability, the docker daemon, `minicloud0` and its routes, port 5000 ownership,
+`USER`/`HOME`, disk headroom - and reports every problem in one pass. `numExecutors=1`, exclusive
+mode and the egress list stay manual agent configuration: nothing in the build can check them.
 
 ### The jobs
 

docs/sct-pipelines.md

@@ -33,3 +33,34 @@ longevityPipeline(
     test_config: 'test-cases/longevity/longevity-10gb-3h.yaml',
 )
 ```
+
+## Running on a local Jenkins agent
+
+By default a pipeline schedules on the region-appropriate cloud builder
+(`getJenkinsLabels` maps backend+region to an ASG/template label), and longevity-style
+pipelines then provision a cloud sct-runner that the test actually runs on. Two knobs change
+that, both usable independently:
+
+- **`jenkins_label` (job parameter, any backend, any pipeline that declares it)** - pin the
+  build to a named agent label, bypassing the region→builder mapping entirely. Only approved
+  labels are accepted: the builder labels `getJenkinsLabels` itself maps to, the
+  `minicloud-kvm-builders-*` family, and the `pinnableExtras` list in that file. A new lab
+  machine becomes pinnable with a one-line addition there. Empty (the default) keeps the
+  normal mapping, so build #1 - where parameters are not yet populated - behaves exactly as
+  before. Use it to debug on a specific machine or to prove out a new agent.
+
+- **`local_agent: true` (jenkinsfile literal, longevity / rolling-upgrade)** - skip the
+  *Create SCT Runner* / *Clean SCT Runners* stages and run the test right on the Jenkins
+  agent, the way `artifactsPipeline` always has (every stage helper already takes the
+  builder-local path when `./sct_runner_ip` is absent). It must be combined with something
+  that picks an appropriate agent - `minicloud: true` (which resolves the KVM-capable label)
+  or a `jenkins_label`.
+
+Both are job parameters as well as jenkinsfile knobs, with one caveat worth knowing: the agent
+label is decided at pipeline-definition time, and `params` is empty on build #1 - the
+parameters-loading build these pipelines abort anyway - so a parameter only takes effect from
+build #2 onward, falling back to the jenkinsfile value before that.
+
+For minicloud specifically - which needs a KVM-capable agent for the local topology - see
+[minicloud](./minicloud.md). For the cloud sct-runner mechanics, see
+[sct-runners](./sct-runners.md).

sdcm/utils/lint/env_builder.py

@@ -163,6 +163,9 @@
         # minicloud_docker maps to SCT_MINICLOUD_DOCKER_IMAGE (the minicloud_docker_image option), but only when minicloud is on -
         # the pipeline decides, so it is not a static jenkinsfile-to-env mapping
         "minicloud_docker",
+        # local_agent picks the topology (KVM Jenkins agent vs sct-runner), which decides which
+        # stages run and which agent label is resolved - pipeline orchestration, not an SCT setting
+        "local_agent",
     }
 )
 

vars/artifactsPipeline.groovy

@@ -6,14 +6,33 @@ def call(Map pipelineParams) {
     // to be decided already. Never read params.minicloud - getJenkinsLabels merges `overrides` into
     // an un-def'd global `params`, so it would look like it works and then not.
     def minicloudEnabled = pipelineParams.get('minicloud', false)
-    // minicloud boots its QEMU/KVM guests on a nested-virtualization sct-runner, sized by
-    // instance_type_runner - see configurations/minicloud/{aws,gce}.yaml. Artifacts jobs had no
-    // runner stage at all before, so this is the whole of it; a regular (non-minicloud)
-    // artifacts run still creates no runner and is byte-identical to before.
-    def useRunner = minicloudEnabled
+    // Same knob and default as longevityPipeline: the nested-virtualization sct-runner stays the
+    // default (sized by instance_type_runner - see configurations/minicloud/{aws,gce}.yaml), and
+    // `local_agent: true` opts a jenkinsfile into a KVM-capable Jenkins agent instead. Opt-in
+    // rather than opt-out on purpose: no node serves the KVM label yet, so a default of true
+    // would strand every minicloud artifacts job in the queue. Only consulted when minicloud is
+    // enabled; a regular artifacts run creates no runner either way.
+    // A build parameter as well as a jenkinsfile knob, so an existing job can be moved to a lab
+    // agent for one run without a new jenkinsfile: tick `local_agent` and the KVM label is
+    // resolved for you. Read off `params` first - it is populated from build #2 onward, which is
+    // where a per-build choice can exist at all - and it falls back to the jenkinsfile default on
+    // build #1, the params-loading build these pipelines abort anyway. Safe to read here, unlike
+    // params.minicloud: getJenkinsLabels only ever merges a `minicloud` key into the global
+    // binding, never this one.
+    // Gated on minicloudEnabled: without minicloud there is nothing to run on a KVM agent, and
+    // an ungated `local_agent` on a normal build would skip the runner stages while the label
+    // stayed the ordinary cloud builder - the test would then run on the builder itself.
+    def localAgent = minicloudEnabled && (params.local_agent != null
+                                          ? params.local_agent.toString().toBoolean()
+                                          : pipelineParams.get('local_agent', false))
+    def useRunner = minicloudEnabled && !localAgent
 
+    // On a local agent minicloud runs its guests on the agent itself, so it needs the KVM-capable
+    // label rather than a region builder. In the default runner topology the build runs on the
+    // usual region builder and the guests live on the sct-runner, like longevity.
     def builder = getJenkinsLabels(params.backend, params.region, params.gce_datacenter,
-                                   params.azure_region_name, params.oci_region_name)
+                                   params.azure_region_name, params.oci_region_name,
+                                   (minicloudEnabled && localAgent) ? [minicloud: true] : null)
 
     pipeline {
         agent none
@@ -104,6 +123,18 @@ def call(Map pipelineParams) {
             // test-case configuration (configurations/minicloud.yaml and
             // configurations/minicloud/*.yaml), not job knobs.
             separator(name: 'MINICLOUD_CONFIG', sectionHeader: 'Minicloud Configuration')
+            booleanParam(defaultValue: "${pipelineParams.get('local_agent', false)}",
+                   description: 'Run minicloud on a KVM-capable Jenkins agent instead of ' +
+                                'provisioning an sct-runner. Needs an agent serving the ' +
+                                'minicloud label (or set jenkins_label). ' +
+                                'Ignored unless the jenkinsfile sets `minicloud: true`',
+                   name: 'local_agent')
+            string(defaultValue: '',
+                   description: 'Pin this build to a specific Jenkins agent label, e.g. a lab ' +
+                                'machine. Empty picks the usual builder for the backend/region, ' +
+                                'or the minicloud label when local_agent is set. A single label, ' +
+                                'not a label expression',
+                   name: 'jenkins_label')
             string(defaultValue: "${pipelineParams.get('minicloud_docker', '')}",
                    description: 'Minicloud Docker image reference, e.g. ghcr.io/scylladb/minicloud:<tag>. ' +
                                 'Empty leaves the image at its renovate-managed default (defaults/docker_images/minicloud/). ' +
@@ -195,7 +226,12 @@ def call(Map pipelineParams) {
                                                 // extra_environment_variables is loaded (a per-run override wins) and
                                                 // before anything talks to a cloud API.
                                                 startMinicloud.exportEnv(params, pipelineParams)
-                                                tagBuilder()
+                                                // Nothing to tag on a local agent: tagBuilder spends 5-8s on DMI
+                                                // reads and four IMDS curls before no-opping with UNKNOWN. The
+                                                // runner topology runs on a regular cloud builder, so tag it.
+                                                if (!(minicloudEnabled && localAgent)) {
+                                                    tagBuilder()
+                                                }
                                             }
                                             dir('scylla-cluster-tests') {
                                                 timeout(time: 10, unit: 'MINUTES') {
@@ -209,6 +245,25 @@ def call(Map pipelineParams) {
                                         // available inside a parallel branch, so try/finally is the only place
                                         // teardown can live.
                                         try {
+                                        if (minicloudEnabled && localAgent) {
+                                            stage("Minicloud Reclaim (${instance_type})") {
+                                                dir('scylla-cluster-tests') {
+                                                    timeout(time: 10, unit: 'MINUTES') {
+                                                        minicloudReclaim()
+                                                    }
+                                                }
+                                            }
+                                            // Fail here, before Argus registration and the hydra pull, rather than
+                                            // 20 minutes in. Not wrapped in catchError: an agent that cannot host
+                                            // minicloud must stop the build, not proceed to talk to the real cloud.
+                                            stage("Minicloud Preflight (${instance_type})") {
+                                                dir('scylla-cluster-tests') {
+                                                    timeout(time: 5, unit: 'MINUTES') {
+                                                        minicloudPreflight()
+                                                    }
+                                                }
+                                            }
+                                        }
                                         if (useRunner) {
                                             // Before Start Minicloud: startMinicloud reads ./sct_runner_ip, which
                                             // create-runner-instance writes, so the container comes up on the
@@ -437,6 +492,24 @@ def call(Map pipelineParams) {
                                                     }
                                                 }
                                             }
+                                            // Last, deliberately: removing the container kills every guest with
+                                            // it (QEMU shares its PID namespace), and 'Collect log data' above
+                                            // needs them alive. A no-op in runner topology, where the container
+                                            // died with the instance the stage above terminated.
+                                            if (minicloudEnabled) {
+                                                stage("Stop Minicloud (${instance_type})") {
+                                                    catchError(stageResult: 'FAILURE') {
+                                                        dir('scylla-cluster-tests') {
+                                                            timeout(time: 10, unit: 'MINUTES') {
+                                                                stopMinicloud(params, currentBuild)
+                                                                // Leave the agent clean for the
+                                                                // next job, minicloud-aware or not.
+                                                                minicloudReclaim(atEnd: true)
+                                                            }
+                                                        }
+                                                    }
+                                                }
+                                            }
                                         }
                                     }
                                 }