diff --git a/docs/minicloud.md b/docs/minicloud.md index a355f25901d..6d6bb57b054 100644 --- a/docs/minicloud.md +++ b/docs/minicloud.md @@ -253,8 +253,44 @@ longevity/rolling-upgrade test-cases set it themselves, and the artifacts jobs l 'Clean SCT Runners' - which takes the container and every guest with it, so it runs after log collection. -Running minicloud on a long-lived KVM Jenkins agent instead is a follow-up; it needs agent -validation, workspace reclaim and a local teardown path that this PR deliberately leaves out. +### The local-agent topology + +`local_agent: true` in the jenkinsfile runs everything on a KVM-capable Jenkins agent instead +(label `minicloud-kvm-builders-v1`), with no sct-runner at all: the runner stages are skipped and +every stage takes the builder-local branch, selected as everywhere else by the absence of +`./sct_runner_ip`. The build gains *Minicloud Reclaim → Minicloud Preflight* ahead of +*Start Minicloud*, and *Stop Minicloud* last - after log collection, because `docker rm -f` kills +every guest with the container. + +Two things only a long-lived agent needs, both by design: + +- **Reclaim at build start, not at the end.** The previous failure's logs stay on the box for + post-mortem, which is one of the few genuine advantages of a static agent. The AMI cache + (`~/.cache/minicloud/amis`) is never swept - it is tens of GiB and tens of minutes to rebuild, + and it is the entire economic case for the agent. +- **Preflight in one pass.** A misconfigured agent reports every problem at once rather than one + per build, and it runs before Argus registration and the hydra pull so a bad agent costs + seconds instead of twenty minutes. + +`local_agent` is a **job parameter** as well as a jenkinsfile knob, so an existing minicloud job +can be moved to a lab agent for a single run without a new jenkinsfile: tick it and the KVM label +is resolved for you. It is read off `params` from build #2 onward - build #1 only loads parameters +and these pipelines abort it - falling back to the jenkinsfile default before that. `jenkins_label` +stays the manual override for pinning any build to an approved agent label on any backend, +independent of minicloud; see [sct-pipelines](./sct-pipelines.md). + +### Agent prerequisites + +A lab agent serving `minicloud-kvm-builders-v1` needs: the agent user in `kvm` and `docker` +groups (restart the agent process after `usermod`, reconnecting is not enough); `minicloud0` +pre-created by a boot-time unit (preferred - no sudo needed at run time) or passwordless sudo; +`USER`/`HOME` set and `$HOME` writable with >=80 GiB free; `numExecutors=1` + exclusive mode, +which is what serialises the host singletons (port 5000, the container name, `minicloud0`); +egress to docker.io, ghcr.io, github.com, amazonaws.com, argus.scylladb.com, +downloads.scylladb.com. The *Minicloud Preflight* stage verifies what a shell on the agent can +see - /dev/kvm writability, the docker daemon, `minicloud0` and its routes, port 5000 ownership, +`USER`/`HOME`, disk headroom - and reports every problem in one pass. `numExecutors=1`, exclusive +mode and the egress list stay manual agent configuration: nothing in the build can check them. ### The jobs diff --git a/docs/sct-pipelines.md b/docs/sct-pipelines.md index 196755da7ee..a251c0a1cbb 100644 --- a/docs/sct-pipelines.md +++ b/docs/sct-pipelines.md @@ -33,3 +33,34 @@ longevityPipeline( test_config: 'test-cases/longevity/longevity-10gb-3h.yaml', ) ``` + +## Running on a local Jenkins agent + +By default a pipeline schedules on the region-appropriate cloud builder +(`getJenkinsLabels` maps backend+region to an ASG/template label), and longevity-style +pipelines then provision a cloud sct-runner that the test actually runs on. Two knobs change +that, both usable independently: + +- **`jenkins_label` (job parameter, any backend, any pipeline that declares it)** - pin the + build to a named agent label, bypassing the region→builder mapping entirely. Only approved + labels are accepted: the builder labels `getJenkinsLabels` itself maps to, the + `minicloud-kvm-builders-*` family, and the `pinnableExtras` list in that file. A new lab + machine becomes pinnable with a one-line addition there. Empty (the default) keeps the + normal mapping, so build #1 - where parameters are not yet populated - behaves exactly as + before. Use it to debug on a specific machine or to prove out a new agent. + +- **`local_agent: true` (jenkinsfile literal, longevity / rolling-upgrade)** - skip the + *Create SCT Runner* / *Clean SCT Runners* stages and run the test right on the Jenkins + agent, the way `artifactsPipeline` always has (every stage helper already takes the + builder-local path when `./sct_runner_ip` is absent). It must be combined with something + that picks an appropriate agent - `minicloud: true` (which resolves the KVM-capable label) + or a `jenkins_label`. + +Both are job parameters as well as jenkinsfile knobs, with one caveat worth knowing: the agent +label is decided at pipeline-definition time, and `params` is empty on build #1 - the +parameters-loading build these pipelines abort anyway - so a parameter only takes effect from +build #2 onward, falling back to the jenkinsfile value before that. + +For minicloud specifically - which needs a KVM-capable agent for the local topology - see +[minicloud](./minicloud.md). For the cloud sct-runner mechanics, see +[sct-runners](./sct-runners.md). diff --git a/sdcm/utils/lint/env_builder.py b/sdcm/utils/lint/env_builder.py index 985a6345313..0563059e82a 100644 --- a/sdcm/utils/lint/env_builder.py +++ b/sdcm/utils/lint/env_builder.py @@ -163,6 +163,9 @@ # minicloud_docker maps to SCT_MINICLOUD_DOCKER_IMAGE (the minicloud_docker_image option), but only when minicloud is on - # the pipeline decides, so it is not a static jenkinsfile-to-env mapping "minicloud_docker", + # local_agent picks the topology (KVM Jenkins agent vs sct-runner), which decides which + # stages run and which agent label is resolved - pipeline orchestration, not an SCT setting + "local_agent", } ) diff --git a/vars/artifactsPipeline.groovy b/vars/artifactsPipeline.groovy index 63b0ce067ca..29dfd53e75c 100644 --- a/vars/artifactsPipeline.groovy +++ b/vars/artifactsPipeline.groovy @@ -6,14 +6,33 @@ def call(Map pipelineParams) { // to be decided already. Never read params.minicloud - getJenkinsLabels merges `overrides` into // an un-def'd global `params`, so it would look like it works and then not. def minicloudEnabled = pipelineParams.get('minicloud', false) - // minicloud boots its QEMU/KVM guests on a nested-virtualization sct-runner, sized by - // instance_type_runner - see configurations/minicloud/{aws,gce}.yaml. Artifacts jobs had no - // runner stage at all before, so this is the whole of it; a regular (non-minicloud) - // artifacts run still creates no runner and is byte-identical to before. - def useRunner = minicloudEnabled + // Same knob and default as longevityPipeline: the nested-virtualization sct-runner stays the + // default (sized by instance_type_runner - see configurations/minicloud/{aws,gce}.yaml), and + // `local_agent: true` opts a jenkinsfile into a KVM-capable Jenkins agent instead. Opt-in + // rather than opt-out on purpose: no node serves the KVM label yet, so a default of true + // would strand every minicloud artifacts job in the queue. Only consulted when minicloud is + // enabled; a regular artifacts run creates no runner either way. + // A build parameter as well as a jenkinsfile knob, so an existing job can be moved to a lab + // agent for one run without a new jenkinsfile: tick `local_agent` and the KVM label is + // resolved for you. Read off `params` first - it is populated from build #2 onward, which is + // where a per-build choice can exist at all - and it falls back to the jenkinsfile default on + // build #1, the params-loading build these pipelines abort anyway. Safe to read here, unlike + // params.minicloud: getJenkinsLabels only ever merges a `minicloud` key into the global + // binding, never this one. + // Gated on minicloudEnabled: without minicloud there is nothing to run on a KVM agent, and + // an ungated `local_agent` on a normal build would skip the runner stages while the label + // stayed the ordinary cloud builder - the test would then run on the builder itself. + def localAgent = minicloudEnabled && (params.local_agent != null + ? params.local_agent.toString().toBoolean() + : pipelineParams.get('local_agent', false)) + def useRunner = minicloudEnabled && !localAgent + // On a local agent minicloud runs its guests on the agent itself, so it needs the KVM-capable + // label rather than a region builder. In the default runner topology the build runs on the + // usual region builder and the guests live on the sct-runner, like longevity. def builder = getJenkinsLabels(params.backend, params.region, params.gce_datacenter, - params.azure_region_name, params.oci_region_name) + params.azure_region_name, params.oci_region_name, + (minicloudEnabled && localAgent) ? [minicloud: true] : null) pipeline { agent none @@ -104,6 +123,18 @@ def call(Map pipelineParams) { // test-case configuration (configurations/minicloud.yaml and // configurations/minicloud/*.yaml), not job knobs. separator(name: 'MINICLOUD_CONFIG', sectionHeader: 'Minicloud Configuration') + booleanParam(defaultValue: "${pipelineParams.get('local_agent', false)}", + description: 'Run minicloud on a KVM-capable Jenkins agent instead of ' + + 'provisioning an sct-runner. Needs an agent serving the ' + + 'minicloud label (or set jenkins_label). ' + + 'Ignored unless the jenkinsfile sets `minicloud: true`', + name: 'local_agent') + string(defaultValue: '', + description: 'Pin this build to a specific Jenkins agent label, e.g. a lab ' + + 'machine. Empty picks the usual builder for the backend/region, ' + + 'or the minicloud label when local_agent is set. A single label, ' + + 'not a label expression', + name: 'jenkins_label') string(defaultValue: "${pipelineParams.get('minicloud_docker', '')}", description: 'Minicloud Docker image reference, e.g. ghcr.io/scylladb/minicloud:. ' + 'Empty leaves the image at its renovate-managed default (defaults/docker_images/minicloud/). ' + @@ -195,7 +226,12 @@ def call(Map pipelineParams) { // extra_environment_variables is loaded (a per-run override wins) and // before anything talks to a cloud API. startMinicloud.exportEnv(params, pipelineParams) - tagBuilder() + // Nothing to tag on a local agent: tagBuilder spends 5-8s on DMI + // reads and four IMDS curls before no-opping with UNKNOWN. The + // runner topology runs on a regular cloud builder, so tag it. + if (!(minicloudEnabled && localAgent)) { + tagBuilder() + } } dir('scylla-cluster-tests') { timeout(time: 10, unit: 'MINUTES') { @@ -209,6 +245,25 @@ def call(Map pipelineParams) { // available inside a parallel branch, so try/finally is the only place // teardown can live. try { + if (minicloudEnabled && localAgent) { + stage("Minicloud Reclaim (${instance_type})") { + dir('scylla-cluster-tests') { + timeout(time: 10, unit: 'MINUTES') { + minicloudReclaim() + } + } + } + // Fail here, before Argus registration and the hydra pull, rather than + // 20 minutes in. Not wrapped in catchError: an agent that cannot host + // minicloud must stop the build, not proceed to talk to the real cloud. + stage("Minicloud Preflight (${instance_type})") { + dir('scylla-cluster-tests') { + timeout(time: 5, unit: 'MINUTES') { + minicloudPreflight() + } + } + } + } if (useRunner) { // Before Start Minicloud: startMinicloud reads ./sct_runner_ip, which // create-runner-instance writes, so the container comes up on the @@ -437,6 +492,24 @@ def call(Map pipelineParams) { } } } + // Last, deliberately: removing the container kills every guest with + // it (QEMU shares its PID namespace), and 'Collect log data' above + // needs them alive. A no-op in runner topology, where the container + // died with the instance the stage above terminated. + if (minicloudEnabled) { + stage("Stop Minicloud (${instance_type})") { + catchError(stageResult: 'FAILURE') { + dir('scylla-cluster-tests') { + timeout(time: 10, unit: 'MINUTES') { + stopMinicloud(params, currentBuild) + // Leave the agent clean for the + // next job, minicloud-aware or not. + minicloudReclaim(atEnd: true) + } + } + } + } + } } } } diff --git a/vars/getJenkinsLabels.groovy b/vars/getJenkinsLabels.groovy index 836b12126b9..ef9e4e20916 100644 --- a/vars/getJenkinsLabels.groovy +++ b/vars/getJenkinsLabels.groovy @@ -8,6 +8,11 @@ def call(String backend, String region=null, String datacenter=null, String loca if (overrides == null){ overrides = [:] } + // NOTE: `params` here has no `def`, so in a shared-library script this writes the build's + // *global* binding and the merge below leaks out of this function. That makes an override + // like `minicloud` *appear* to work as `params.minicloud` in a later `when {}` guard - do + // not rely on it. Guards elsewhere must read a captured local. Left as-is rather than + // localised because callers have depended on the leak for years. params += overrides // merge, overrides take precedence if (!backend) { @@ -62,6 +67,7 @@ def call(String backend, String region=null, String datacenter=null, String loca 'oci-us-ashburn-1': 'oci-sct-builders-us-ashburn-1-v2', 'oci-us-phoenix-1': 'oci-sct-builders-us-phoenix-1-v2', 'oci-eu-frankfurt-1': 'oci-sct-builders-eu-frankfurt-1-v2', + 'minicloud': 'minicloud-kvm-builders-v1', ] def cloud_provider = getCloudProviderFromBackend(backend) @@ -71,26 +77,89 @@ def call(String backend, String region=null, String datacenter=null, String loca cloud_provider = params.xcloud_provider?.trim()?.toLowerCase() } - if ((cloud_provider == 'aws' && region) || (cloud_provider == 'gce' && datacenter) || (cloud_provider == 'azure' && location) || (cloud_provider == 'aws-fibs' && region) || (cloud_provider == 'oci' && oci_region)) { - def supported_regions = [] - - if (cloud_provider == 'aws') { - supported_regions = ["eu-west-2", "eu-north-1", "eu-central-1", "us-west-2", "us-east-2", "eu-west-3", "ca-central-1"] - } else if (cloud_provider == 'gce') { - supported_regions = ["us-east1", "us-east4", "us-west1", "us-central1"] - region = datacenter - } else if (cloud_provider == 'azure') { - supported_regions = ["eastus"] - region = location - } else if (cloud_provider == 'oci') { - supported_regions = ["us-ashburn-1", "us-phoenix-1", "eu-frankfurt-1"] - region = oci_region + def supported_regions_by_provider = [ + 'aws': ['eu-west-2', 'eu-north-1', 'eu-central-1', 'us-west-2', 'us-east-2', 'eu-west-3', 'ca-central-1'], + 'gce': ['us-east1', 'us-east4', 'us-west1', 'us-central1'], + 'azure': ['eastus'], + 'oci': ['us-ashburn-1', 'us-phoenix-1', 'eu-frankfurt-1'], + ] + + // The effective backend region, shared by every return below: GCE/Azure/OCI callers pass + // their region in datacenter/location/oci_region while `region` stays null. The pipelines + // hand the returned value straight to createSctRunner and the collect/clean stages, which + // act on the real cloud with it, so it must be a real region for every backend - the label + // alone is never enough. + def effectiveRegion = region + if (cloud_provider == 'gce') { + effectiveRegion = datacenter + } else if (cloud_provider == 'azure') { + effectiveRegion = location + } else if (cloud_provider == 'oci') { + effectiveRegion = oci_region + } + + // Both early returns below deliberately sit after the JSON-list unwrapping (so `region` is a + // plain string) and before the region->builder mapping: a build that runs on a local agent has + // no region builder to find, and for minicloud the region is the one it *emulates*. + + // Pin this build to a named Jenkins agent, on any backend. Declaring the `jenkins_label` + // parameter in a pipeline is the entire integration - this function already reads the global + // `params` binding. Empty on build #1, where params is not yet populated, so the normal + // mapping applies there and nothing changes for the ~1100 jobs that never set it. + def pinnedLabel = params.jenkins_label?.trim() + if (pinnedLabel) { + // A single agent label, never a label *expression*: Jenkins would happily evaluate + // `a||b` here, so a free-form value lets whoever can trigger the job aim it at agents + // that were never meant to run SCT - with this job's credentials bound. + if (!(pinnedLabel ==~ /^[\w][\w.\-]*$/)) { + error("jenkins_label must be a single agent label (letters, digits, dot, dash, " + + "underscore), not a label expression - got '${pinnedLabel}'") + } + // and only a label this repository approves: everything this function can already map + // to, the minicloud KVM agent family, or `pinnableExtras`. A machine becomes pinnable + // by a one-line addition here, so there is a review trail for "this host may run SCT + // jobs, with their credentials". Individual lab machines go into the extras list. + def pinnableExtras = [] + def pinnable = new ArrayList(pinnableExtras) + for (v in jenkins_labels.values()) { + pinnable.add(v.toString()) // GCE entries are GStrings - normalize before compare + } + if (!pinnable.contains(pinnedLabel) && !(pinnedLabel ==~ /^minicloud-kvm-builders-[\w.\-]+$/)) { + error("jenkins_label '${pinnedLabel}' is not an approved SCT agent label: use a " + + "builder label from getJenkinsLabels.groovy, a minicloud-kvm-builders-* label, " + + "or add the new label to pinnableExtras in that file") } + if (effectiveRegion == 'random') { + def choices = new ArrayList(supported_regions_by_provider.get(cloud_provider, ['eu-west-1'])) + Collections.shuffle(choices) + effectiveRegion = choices[0] + } + println("Pinned to Jenkins agent label: " + pinnedLabel + ", region: " + (effectiveRegion ?: 'eu-west-1')) + return [ "label": pinnedLabel, "region": effectiveRegion ?: 'eu-west-1' ] + } + + // minicloud boots QEMU/KVM guests on the agent itself, so it needs a KVM-capable node rather + // than a region builder. Comes through `overrides` rather than `params` because there is no + // params.minicloud on build #1, which is when the agent label is first evaluated. + if (overrides.minicloud) { + if (effectiveRegion == 'random') { + throw new Exception("=================== minicloud needs a fixed region, not 'random': it " + + "validates uncached AMIs against its own --aws-region ===================") + } + def minicloudLabel = jenkins_labels['minicloud'] + println("minicloud run: using KVM-capable agent label " + minicloudLabel) + return [ "label": minicloudLabel, "region": effectiveRegion ?: 'eu-west-1' ] + } + + if (effectiveRegion && cloud_provider in ['aws', 'gce', 'azure', 'aws-fibs', 'oci']) { + def supported_regions = supported_regions_by_provider.get(cloud_provider, []) + region = effectiveRegion println("Finding builder for region: " + region) - if (region == "random" || datacenter == "random" || location == "random") { - Collections.shuffle(supported_regions) - region = supported_regions[0] + if (region == "random") { + def choices = new ArrayList(supported_regions) + Collections.shuffle(choices) + region = choices[0] } def cp_region = cloud_provider + "-" + region diff --git a/vars/longevityPipeline.groovy b/vars/longevityPipeline.groovy index 4ebcde586c1..6035e662501 100644 --- a/vars/longevityPipeline.groovy +++ b/vars/longevityPipeline.groovy @@ -10,9 +10,27 @@ def call(Map pipelineParams) { // getJenkinsLabels merges its `overrides` into an un-def'd global `params`, so it would appear // to work and then not. def minicloudEnabled = pipelineParams.get('minicloud', false) + // Opt in with `local_agent: true` to run on a Jenkins agent instead of provisioning an + // sct-runner. Unlike artifacts, longevity keeps both topologies: a nested-virtualization cloud + // runner (sized by instance_type_runner) is still the default, and remains the only option for + // anyone without access to a KVM-capable lab agent. + // A build parameter as well as a jenkinsfile knob, so an existing job can be moved to a lab + // agent for one run without a new jenkinsfile: tick `local_agent` and the KVM label is + // resolved for you. Read off `params` first - it is populated from build #2 onward, which is + // where a per-build choice can exist at all - and it falls back to the jenkinsfile default on + // build #1, the params-loading build these pipelines abort anyway. Safe to read here, unlike + // params.minicloud: getJenkinsLabels only ever merges a `minicloud` key into the global + // binding, never this one. + // Gated on minicloudEnabled: without minicloud there is nothing to run on a KVM agent, and + // an ungated `local_agent` on a normal build would skip the runner stages while the label + // stayed the ordinary cloud builder - the test would then run on the builder itself. + def localAgent = minicloudEnabled && (params.local_agent != null + ? params.local_agent.toString().toBoolean() + : pipelineParams.get('local_agent', false)) def builder = getJenkinsLabels(params.backend, params.region, params.gce_datacenter, - params.azure_region_name, params.oci_region_name) + params.azure_region_name, params.oci_region_name, + (minicloudEnabled && localAgent) ? [minicloud: true] : null) def functional_test = pipelineParams.functional_test pipeline { @@ -103,6 +121,12 @@ def call(Map pipelineParams) { // vars/startMinicloud.groovy. Everything else minicloud needs - KMS off, a // KVM-capable instance_type_runner - is test-case configuration, not a job knob. separator(name: 'MINICLOUD_CONFIG', sectionHeader: 'Minicloud Configuration') + booleanParam(defaultValue: "${pipelineParams.get('local_agent', false)}", + description: 'Run minicloud on a KVM-capable Jenkins agent instead of ' + + 'provisioning an sct-runner. Needs an agent serving the ' + + 'minicloud label (or set jenkins_label). ' + + 'Ignored unless the jenkinsfile sets `minicloud: true`', + name: 'local_agent') string(defaultValue: "${pipelineParams.get('minicloud_docker', '')}", description: 'Minicloud Docker image reference, e.g. ghcr.io/scylladb/minicloud:. ' + 'Empty leaves the image at its renovate-managed default (defaults/docker_images/minicloud/). ' + @@ -229,6 +253,10 @@ def call(Map pipelineParams) { string(defaultValue: "${pipelineParams.get('gce_project', '')}", description: 'Gce project to use', name: 'gce_project') + string(defaultValue: '', + description: 'Pin this build to a specific Jenkins agent label, e.g. a lab machine. ' + + 'Empty picks the usual builder for the backend/region', + name: 'jenkins_label') string(defaultValue: '', description: 'Actual user requesting job start, for automated job builds (e.g. through Argus)', name: 'requested_by_user') @@ -310,6 +338,25 @@ def call(Map pipelineParams) { dockerLogin(params) } } + // Only on a local agent: a persistent workspace needs sweeping, and the agent has to be + // proven KVM-capable, before anything expensive or externally visible happens - so this + // sits ahead of 'Create Argus Test Run' and the hydra pull it triggers, not after them. + // Reclaim is also the only thing that removes a stale ./sct_runner_ip once 'Create SCT + // Runner' is skipped; without it every later stage would take the --execute-on-runner + // branch and SSH to an IP from some earlier build. + stage('Minicloud Agent Preflight') { + when { expression { minicloudEnabled && localAgent } } + steps { + script { + dir('scylla-cluster-tests') { + timeout(time: 10, unit: 'MINUTES') { + minicloudReclaim() + minicloudPreflight() + } + } + } + } + } stage('Create Argus Test Run') { steps { catchError(stageResult: 'FAILURE') { @@ -368,6 +415,8 @@ def call(Map pipelineParams) { } } stage('Create SCT Runner') { + // On a local agent the test runs right here, so there is nothing to provision. + when { expression { !localAgent } } steps { script { wrap([$class: 'BuildUser']) { @@ -496,6 +545,8 @@ def call(Map pipelineParams) { } } stage('Clean SCT Runners') { + // Nothing was provisioned on a local agent, so there is nothing to reclaim. + when { expression { !localAgent } } steps { catchError(stageResult: 'FAILURE') { script { @@ -581,7 +632,10 @@ def call(Map pipelineParams) { } } } - if (!completed_stages['clean_sct_runner']) { + // `!localAgent` and not just the completed_stages check: the stage above is + // skipped by its when{}, which leaves this recovery path as the one thing that + // would still go looking for a runner that was never created. + if (!localAgent && !completed_stages['clean_sct_runner']) { catchError { script { wrap([$class: 'BuildUser']) { @@ -592,6 +646,22 @@ def call(Map pipelineParams) { } } } + // Deliberately the last thing in post{}: `docker rm -f` kills every guest with + // the container, and the log collection above needs them alive. + if (minicloudEnabled) { + catchError { + script { + dir('scylla-cluster-tests') { + timeout(time: 10, unit: 'MINUTES') { + stopMinicloud(params, currentBuild) + // Leave the agent clean for the next job, which may be a + // scylla build or dtest and will not clean up after us. + minicloudReclaim(atEnd: true) + } + } + } + } + } } } } diff --git a/vars/minicloudPreflight.groovy b/vars/minicloudPreflight.groovy new file mode 100644 index 00000000000..8566b5a1abb --- /dev/null +++ b/vars/minicloudPreflight.groovy @@ -0,0 +1,157 @@ +#!groovy + +// Validate that this Jenkins agent can actually host minicloud, before anything expensive runs. +// +// Every probe is bounded and nothing throws mid-way: failures are collected and reported once at +// the end, so a misconfigured agent costs one build to diagnose rather than one build per problem. +// Same shape as tagBuilder.groovy's identifyCloud(). +// +// Everything probed here is a property of the AGENT, not of the run, which is why it is not simply +// a call into MinicloudManager.preflight_check(): that one runs inside the test process, ~20 minutes +// in, after Argus registration, the hydra pull and the checkout, from inside a container - so it +// cannot answer any of the questions below. Where the two do overlap the Python side is the one +// kept (see the RAM note further down); nothing here re-implements a check Python can make. +// +// What Python cannot see from where it runs: +// +// * it does Path("/dev/kvm").exists(), which says nothing about whether the agent user can +// *open* it. That is the single most likely misconfiguration on a shared lab box. +// * it does shutil.which("docker"), which says nothing about the daemon being reachable or the +// agent user being in the docker group. +// +// Exact RAM sizing is deliberately NOT here - it belongs in Python, where SCTConfiguration knows +// n_db_nodes + n_loaders + n_monitor_nodes after the yaml+env merge and can parse IntOrList +// ("3 3" for multi-DC). See MinicloudManager._check_host_memory(). +// +// Nested virtualization is not checked: these agents are bare metal, so KVM is native. + +def call(Map args = [:]) { + def minHomeGib = args.get('minHomeGib', 80) + def minWorkspaceGib = args.get('minWorkspaceGib', 20) + + // One shell, one report. `set +e` throughout: a probe that cannot run is a finding, not a + // reason to abort before the other probes have said their piece. + def report = sh(returnStdout: true, script: """#!/bin/bash +set +e + +fail=() +warn=() + +# --- KVM ----------------------------------------------------------------------------------- +if [[ ! -e /dev/kvm ]] ; then + fail+=("/dev/kvm is missing - this agent cannot run minicloud guests") +elif [[ ! -w /dev/kvm ]] ; then + fail+=("/dev/kvm exists but is not writable by \$(id -un) (groups: \$(id -Gn)) - add the agent user to the 'kvm' group and RESTART the agent process, reconnecting is not enough") +fi + +if ! grep -qE '^(kvm_intel|kvm_amd) ' /proc/modules ; then + # -n so this can never sit waiting for a password prompt + if ! sudo -n modprobe kvm_intel 2>/dev/null && ! sudo -n modprobe kvm_amd 2>/dev/null ; then + fail+=("no kvm_intel/kvm_amd module loaded and it could not be modprobe'd without a password") + fi +fi + +# --- docker -------------------------------------------------------------------------------- +if ! command -v docker >/dev/null 2>&1 ; then + fail+=("docker is not on PATH") +elif ! timeout 30 docker info >/dev/null 2>&1 ; then + fail+=("docker is on PATH but 'docker info' failed - daemon down, or \$(id -un) is not in the 'docker' group") +fi + +# --- host networking ------------------------------------------------------------------------ +# _setup_host_networking() only LOGGER.warning's when it cannot create minicloud0, so without one +# of these two the build proceeds with every guest silently unreachable. +# Full `inet /` token, not a substring: '10.127.0.1' also matches 10.127.0.10/24, +# which would pass here and then leave every guest unreachable. Mirrors _host_networking_matches(). +if ! ip addr show minicloud0 2>/dev/null | grep -qE 'inet 10\\.127\\.0\\.1/24( |\$)' ; then + if ! sudo -n true 2>/dev/null ; then + fail+=("minicloud0 does not carry 10.127.0.1/24 and there is no passwordless sudo to create it - either pre-create it with a boot-time unit (preferred) or grant sudo -n") + fi +else + # The device exists, so nothing later will reconfigure it: the routes have to be right now. + # A leftover blanket 10.0.0.0/8 from an older image black-holes Argus and the QA infra. + for want in 10.160.0.0/11 172.31.0.0/16 ; do + if ! ip route show "\${want}" 2>/dev/null | grep -q 'dev minicloud0' ; then + fail+=("minicloud0 exists but \${want} is not routed through it - re-run minicloud-setup.sh with MINICLOUD_VPC_ROUTES, or grant sudo -n so the run can") + fi + done + if ip route show 10.0.0.0/8 2>/dev/null | grep -q 'dev minicloud0' ; then + fail+=("minicloud0 still carries the legacy 10.0.0.0/8 route, which black-holes Argus and the QA infra from this agent - re-run minicloud-setup.sh to replace it") + fi +fi + +# --- port 5000 ----------------------------------------------------------------------------- +if ss -ltn 2>/dev/null | grep -q ':5000 ' ; then + # -f and a body match, both needed: bare `curl -s -o /dev/null` exits 0 on a 404, so any random + # listener on 5000 would pass for a minicloud. DescribeVpcs is also the health check + # MinicloudManager uses - DescribeRegions answers before the API is really ready. + if ! docker ps --filter 'name=^minicloud\$' --format '{{.Names}}' 2>/dev/null | grep -qx minicloud ; then + # Answering the probe is not enough: it has to be OUR container. Otherwise + # MinicloudManager.start() finds a healthy endpoint it does not own and reuses it. + fail+=("port 5000 is in use but no container named 'minicloud' is running - something else owns it") + elif timeout 10 curl -fs "http://localhost:5000/?Action=DescribeVpcs&Version=2016-11-15" 2>/dev/null | grep -q 'DescribeVpcsResponse' ; then + warn+=("port 5000 is already held by a responding minicloud container - it will be reused or replaced") + else + fail+=("a container named 'minicloud' holds port 5000 but does not answer DescribeVpcs - it is wedged; remove it with 'docker rm -f minicloud'") + fi +fi + +# --- environment --------------------------------------------------------------------------- +# hydra.sh does `id -u "\${USER}"` under `set -eo pipefail`, so an unset USER on a +# systemd-launched JNLP agent kills hydra with a bare "id: '': no such user". +[[ -z "\${USER}" ]] && fail+=("USER is not set - hydra.sh resolves the agent user under 'set -eo pipefail' and dies when it is empty. Common on systemd-launched JNLP agents.") +[[ -z "\${HOME}" ]] && fail+=("HOME is not set") +if [[ -n "\${HOME}" && ! -w "\${HOME}" ]] ; then + fail+=("HOME (\${HOME}) is not writable - the ~/.cache/minicloud AMI cache lives there") +fi + +# --- disk ---------------------------------------------------------------------------------- +# The AMI cache is tens of GiB per image and is the whole economic case for a long-lived agent. +home_free=\$(df --output=avail -BG "\${HOME:-/}" 2>/dev/null | tail -1 | tr -dc '0-9') +if [[ -n "\${home_free}" && "\${home_free}" -lt ${minHomeGib} ]] ; then + fail+=("only \${home_free}GiB free in \${HOME} - need ${minHomeGib}GiB for the minicloud image cache") +fi +ws_free=\$(df --output=avail -BG . 2>/dev/null | tail -1 | tr -dc '0-9') +if [[ -n "\${ws_free}" && "\${ws_free}" -lt ${minWorkspaceGib} ]] ; then + fail+=("only \${ws_free}GiB free in the workspace - need ${minWorkspaceGib}GiB for logs and guest images") +fi + +# --- warn-only ----------------------------------------------------------------------------- +if docker ps -a --format '{{.Names}}' 2>/dev/null | grep -qx minicloud ; then + warn+=("a container named 'minicloud' already exists - it will be replaced") +fi +stale_qemu=\$(pgrep -c qemu-system-x86_64 2>/dev/null) +if [[ -n "\${stale_qemu}" && "\${stale_qemu}" -gt 0 ]] ; then + warn+=("\${stale_qemu} qemu-system-x86_64 process(es) already running - they eat the RAM this build needs, and on a shared box may not be ours") +fi +mem_avail=\$(awk '/MemAvailable/ {print int(\$2/1048576)}' /proc/meminfo 2>/dev/null) +if [[ -n "\${mem_avail}" && "\${mem_avail}" -lt 8 ]] ; then + warn+=("only \${mem_avail}GiB MemAvailable - exact sizing is checked later against the test shape") +fi + +for w in "\${warn[@]}" ; do echo "PREFLIGHT-WARN \${w}" ; done +for f in "\${fail[@]}" ; do echo "PREFLIGHT-FAIL \${f}" ; done +echo "PREFLIGHT-DONE \${#fail[@]}" +""") + + def failures = [] + report.readLines().each { line -> + line = line.trim() + if (line.startsWith('PREFLIGHT-WARN ')) { + println("minicloud preflight WARNING: " + line.substring('PREFLIGHT-WARN '.length())) + } else if (line.startsWith('PREFLIGHT-FAIL ')) { + failures << line.substring('PREFLIGHT-FAIL '.length()) + } + } + + if (failures) { + def message = "=================== This agent cannot host minicloud (${failures.size()} problem(s)) " + + "===================\n" + failures.collect { " * ${it}" }.join('\n') + + "\nSee docs/minicloud.md for the agent prerequisites." + println(message) + throw new Exception(message) + } + + println("minicloud preflight passed on ${env.NODE_NAME}") + return true +} diff --git a/vars/minicloudReclaim.groovy b/vars/minicloudReclaim.groovy new file mode 100644 index 00000000000..90582325714 --- /dev/null +++ b/vars/minicloudReclaim.groovy @@ -0,0 +1,80 @@ +#!groovy + +// Reclaim disk on a long-lived Jenkins agent running minicloud builds. +// +// Cloud builders die after their build, so nothing in SCT cleans a workspace up: `--logdir $(pwd)` +// leaves a / tree behind, plus ./latest and ~/.cache/minicloud/{instances,amis}. On an +// agent that lives for months that accumulates until the disk fills. +// +// Called TWICE per build, deliberately: +// +// minicloudReclaim() at build start - sweeps what earlier builds left, and clears a +// stale ./sct_runner_ip before any stage can act on it +// minicloudReclaim(atEnd: true) at build end - this build's own guest state and container, so +// the agent is left clean for whoever runs next. A shared agent +// also serves scylla builds and dtest, and those jobs know +// nothing about minicloud, so they will never clean up after it. +// +// The end-of-build pass deliberately keeps the log tree: collect-logs has already uploaded it to +// S3/Argus, but leaving it on the box for a few days is the one real advantage a static agent has +// for post-mortem. The start-of-build sweep ages it out. +// +// What is deliberately NOT touched: +// docker images the host is shared with scylla builds and dtest and the RelEng team manage +// image retention their own way; not even a dangling-only prune here, since +// "dangling" includes layers those jobs are mid-way through building +// minicloud0 the host TUN device; recreating it needs sudo we would rather not have +def call(Map args = [:]) { + // Log trees and guest state age out after a few days; the AMI/image cache gets a much longer + // TTL because rebuilding one entry is tens of minutes and tens of GiB - it is the entire + // economic case for a static agent. It still needs a TTL: master images are rebuilt daily, so + // a cache that only ever grows fills the disk on its own. + def keepDays = args.get('keepDays', 3) + def keepImageDays = args.get('keepImageDays', 30) + def atEnd = args.get('atEnd', false) + + sh """#!/bin/bash +# Reclaiming is best-effort: a build must never fail because an old file could not be removed. +set +e +set -x + +if [[ "${atEnd}" == "true" ]] ; then + # This build's own leftovers. The container first: it pins the qcow2 overlays under + # instances/, so removing it before them is what actually frees the disk. + docker ps -a --filter 'name=minicloud' --format '{{.Names}} {{.Status}}' 2>/dev/null + docker rm -f minicloud 2>/dev/null + rm -rf "\${HOME}/.cache/minicloud/instances"/* 2>/dev/null + rm -fv ./sct_runner_ip +else + # Old per-test log trees in the persistent workspace, and the symlink into the newest one. + find . -maxdepth 1 -type d -name '????????-????-????-????-????????????' -mtime +${keepDays} -print -exec rm -rf {} + 2>/dev/null + find . -maxdepth 1 -name 'latest' -type l -delete 2>/dev/null + + # Guest state an aborted build never got to clean. + if [[ -d "\${HOME}/.cache/minicloud/instances" ]] ; then + find "\${HOME}/.cache/minicloud/instances" -maxdepth 1 -mindepth 1 -mtime +${keepDays} -print -exec rm -rf {} + 2>/dev/null + fi + + # The image cache, on its own long TTL - see keepImageDays. Entries are per Scylla + # image/AMI, so a daily master build leaves one behind every day. + if [[ -d "\${HOME}/.cache/minicloud/amis" ]] ; then + find "\${HOME}/.cache/minicloud/amis" -maxdepth 1 -mindepth 1 -mtime +${keepImageDays} -print -exec rm -rf {} + 2>/dev/null + fi + + # A stale sct_runner_ip is actively dangerous here: on a persistent workspace it would send + # every stage down the --execute-on-runner branch and SSH to an IP that belongs to a + # long-dead runner. + rm -fv ./sct_runner_ip + + # A container left by an aborted build. Removed by NAME, never via `docker container prune`: + # the host is explicitly shared, and a host-wide prune would take other jobs' stopped + # containers with it. + docker ps -a --filter 'name=minicloud' --format '{{.Names}} {{.Status}}' 2>/dev/null + docker rm -f minicloud 2>/dev/null +fi + +echo "--- free space after reclaim ---" +df -h "\${HOME}" . 2>/dev/null +exit 0 +""" +} diff --git a/vars/rollingUpgradePipeline.groovy b/vars/rollingUpgradePipeline.groovy index 74a55fad71d..aa25921c9cb 100644 --- a/vars/rollingUpgradePipeline.groovy +++ b/vars/rollingUpgradePipeline.groovy @@ -9,9 +9,25 @@ def call(Map pipelineParams) { // Captured locals - see the same block in longevityPipeline for why these come from // pipelineParams and must never be read back off `params` in a guard. def minicloudEnabled = pipelineParams.get('minicloud', false) + // Rolling upgrade keeps both topologies, like longevity: a nested-virtualization cloud + // sct-runner by default, a KVM-capable Jenkins agent with `local_agent: true`. + // A build parameter as well as a jenkinsfile knob, so an existing job can be moved to a lab + // agent for one run without a new jenkinsfile: tick `local_agent` and the KVM label is + // resolved for you. Read off `params` first - it is populated from build #2 onward, which is + // where a per-build choice can exist at all - and it falls back to the jenkinsfile default on + // build #1, the params-loading build these pipelines abort anyway. Safe to read here, unlike + // params.minicloud: getJenkinsLabels only ever merges a `minicloud` key into the global + // binding, never this one. + // Gated on minicloudEnabled: without minicloud there is nothing to run on a KVM agent, and + // an ungated `local_agent` on a normal build would skip the runner stages while the label + // stayed the ordinary cloud builder - the test would then run on the builder itself. + def localAgent = minicloudEnabled && (params.local_agent != null + ? params.local_agent.toString().toBoolean() + : pipelineParams.get('local_agent', false)) def builder = getJenkinsLabels(params.backend, params.region, params.gce_datacenter, - params.azure_region_name, params.oci_region_name) + params.azure_region_name, params.oci_region_name, + (minicloudEnabled && localAgent) ? [minicloud: true] : null) // since this is a boolean param, we need to handle its default value upfront, we can't do it in the parameters section // we'll keep it as boolean to simplify its usage later on @@ -68,6 +84,12 @@ def call(Map pipelineParams) { // vars/startMinicloud.groovy. Everything else minicloud needs - KMS off, a // KVM-capable instance_type_runner - is test-case configuration, not a job knob. separator(name: 'MINICLOUD_CONFIG', sectionHeader: 'Minicloud Configuration') + booleanParam(defaultValue: "${pipelineParams.get('local_agent', false)}", + description: 'Run minicloud on a KVM-capable Jenkins agent instead of ' + + 'provisioning an sct-runner. Needs an agent serving the ' + + 'minicloud label (or set jenkins_label). ' + + 'Ignored unless the jenkinsfile sets `minicloud: true`', + name: 'local_agent') string(defaultValue: "${pipelineParams.get('minicloud_docker', '')}", description: 'Minicloud Docker image reference, e.g. ghcr.io/scylladb/minicloud:. ' + 'Empty leaves the image at its renovate-managed default (defaults/docker_images/minicloud/). ' + @@ -84,6 +106,10 @@ def call(Map pipelineParams) { description: 'Cap the minicloud container itself, e.g. 32GiB. Empty means no docker limit. ' + 'When set, this is also the budget the preflight guest-memory gate measures against', name: 'minicloud_container_memory') + string(defaultValue: '', + description: 'Pin this build to a specific Jenkins agent label, e.g. a lab machine. ' + + 'Empty picks the usual builder for the backend/region', + name: 'jenkins_label') separator(name: 'POST_BEHAVIOR', sectionHeader: 'Post Behavior Configuration') string(defaultValue: "${pipelineParams.get('post_behavior_db_nodes', 'destroy')}", description: 'keep|keep-on-failure|destroy', @@ -295,6 +321,12 @@ def call(Map pipelineParams) { } } } + stage("Minicloud Agent Preflight for ${base_version}") { + dir('scylla-cluster-tests') { + timeout(time: 10, unit: 'MINUTES') { + minicloudReclaim() + minicloudPreflight() + } stage('Create Argus Test Run') { catchError(stageResult: 'FAILURE') { script { @@ -308,14 +340,23 @@ def call(Map pipelineParams) { } } } - stage("Create SCT Runner for ${base_version}") { - wrap([$class: 'BuildUser']) { - dir('scylla-cluster-tests') { - timeout(time: 5, unit: 'MINUTES') { - createSctRunner(params_mapping[base_version], runnerTimeout, builder.region) + // On a local agent the test runs right here, so there is + // nothing to provision - and minicloudReclaim is then the + // only thing clearing a stale ./sct_runner_ip out of a + // persistent workspace. + if (!localAgent) { + stage("Create SCT Runner for ${base_version}") { + wrap([$class: 'BuildUser']) { + dir('scylla-cluster-tests') { + timeout(time: 5, unit: 'MINUTES') { + createSctRunner(params_mapping[base_version], runnerTimeout, builder.region) + } } } } + } else if (minicloudEnabled) { + } + } } // Must run before 'Provision Resources': that stage calls the // EC2/GCE API, which in minicloud mode means localhost:5000 on @@ -411,12 +452,16 @@ def call(Map pipelineParams) { } } } - stage('Clean SCT Runners') { - catchError(stageResult: 'FAILURE') { - wrap([$class: 'BuildUser']) { - dir('scylla-cluster-tests') { - cleanSctRunners(params_mapping[base_version], currentBuild) - completed_stages[base_version]['clean_sct_runner'] = true + // Nothing was provisioned on a local agent, so there is + // nothing to reclaim. + if (!localAgent) { + stage('Clean SCT Runners') { + catchError(stageResult: 'FAILURE') { + wrap([$class: 'BuildUser']) { + dir('scylla-cluster-tests') { + cleanSctRunners(params_mapping[base_version], currentBuild) + completed_stages[base_version]['clean_sct_runner'] = true + } } } } @@ -488,7 +533,10 @@ def call(Map pipelineParams) { } } } - if (!completed_stages[base_version]['clean_sct_runner']) { + // `!localAgent` and not just the completed_stages check: the + // stage above is skipped on a local agent, which would leave + // this recovery path hunting for a runner that never existed. + if (!localAgent && !completed_stages[base_version]['clean_sct_runner']) { catchError { script { wrap([$class: 'BuildUser']) { @@ -499,6 +547,23 @@ def call(Map pipelineParams) { } } } + // Deliberately the last teardown step: `docker rm -f` kills + // every guest with the container, and the log collection + // above needs them alive. + if (minicloudEnabled) { + catchError { + script { + dir('scylla-cluster-tests') { + timeout(time: 10, unit: 'MINUTES') { + stopMinicloud(params_mapping[base_version], currentBuild) + // Leave the agent clean for + // the next job on it. + minicloudReclaim(atEnd: true) + } + } + } + } + } } } } diff --git a/vars/stopMinicloud.groovy b/vars/stopMinicloud.groovy new file mode 100644 index 00000000000..f82a7a05401 --- /dev/null +++ b/vars/stopMinicloud.groovy @@ -0,0 +1,73 @@ +#!groovy + +// Tear the minicloud container down at the end of a build, and keep its log. +// +// LOCAL-AGENT TOPOLOGY ONLY, by design: in runner topology the container lives on the +// sct-runner and dies with it when 'Clean SCT Runners' terminates the instance - its logs are +// collected earlier by the regular collect-logs stage, which executes on the runner. This +// helper inspects the local docker daemon and is a deliberate no-op there (guarded on +// ./sct_runner_ip below), so callers can keep it unconditional in their `finally`. +// +// ORDERING IS LOAD-BEARING: this must be the *last* teardown step, after log collection. The +// container runs with --network host but not --pid host, so QEMU shares its PID namespace and +// `docker rm -f` kills every guest with it - collect_minicloud_logs() needs them alive. +// +// Honours the post_behavior_* keep modes, because on minicloud the container IS the cluster: +// `docker rm -f` takes every guest with it, so removing it while 'Clean resources' has just +// reported the nodes preserved would make keep/keep-on-failure silently untrue. Same rule as +// cleanSctRunners (all three post_behavior_* set to keep), plus keep-on-failure on a build that +// did not succeed - the case those modes exist for. +// +// Always safe to call: it is a no-op when there is no container, so callers can put it in a +// `finally` without guarding. +def call(Map params = [:], Object build = null) { + def keepAlways = ['db_nodes', 'loader_nodes', 'monitor_nodes'].every { + params."post_behavior_${it}" == 'keep' + } + def buildFailed = build != null && build.currentResult != 'SUCCESS' + def keepOnFailure = buildFailed && ['db_nodes', 'loader_nodes', 'monitor_nodes'].any { + params."post_behavior_${it}" == 'keep-on-failure' + } + def keepGuests = keepAlways || keepOnFailure + + sh """#!/bin/bash +# Teardown must never fail a build that has already produced its result. +set +e +set -x + +# Runner topology: the container is on the runner, not on this builder - nothing to do here. +if [[ -n "\$(cat sct_runner_ip 2>/dev/null)" ]] ; then + echo "runner topology: the minicloud container lives on the sct-runner and is torn down with it" + exit 0 +fi + +if ! docker ps -a --format '{{.Names}}' 2>/dev/null | grep -qx minicloud ; then + echo "no minicloud container to stop" + exit 0 +fi + +# Capture the log before removing the container - this is the only place the emulator's own view of +# the run survives, and it is what tells you whether guests failed to boot or the API rejected a +# call. Bounded: a long run can produce a lot. +mkdir -p ./minicloud-logs +docker logs --tail 5000 minicloud > ./minicloud-logs/minicloud-container.log 2>&1 +echo "--- last 50 lines of the minicloud container log ---" +tail -50 ./minicloud-logs/minicloud-container.log + +if [[ "${keepGuests}" == "true" ]] ; then + # The container is the cluster here: removing it would destroy the very nodes the + # post_behavior_* settings asked to keep. Left running, and left for the next build's + # minicloudReclaim to collect - so "keep" on a shared agent means "until the next build". + echo "post_behavior_* asks to keep the nodes: leaving the minicloud container and its guests running" + echo "inspect with: docker logs minicloud ; ssh into the guests from this agent" + exit 0 +fi + +# Kills the emulator and every guest inside it. +docker rm -f minicloud + +# Leave ~/.cache/minicloud/amis alone: it is the expensive, reusable part. See minicloudReclaim. +exit 0 +""" + archiveArtifacts artifacts: 'minicloud-logs/**', allowEmptyArchive: true +}