fix(integration-tests): run the integration suite in parallel

[fruch]

Refs SCT-804, blocks SCT-714. Sibling of #15691 (same change for branch-2025.1).

Problem

The integration tests stage on branch-perf-v17 runs single-threaded and therefore always exceeds its 45-minute timeout. Jenkins ABORTs the build before pytest can write integration-tests-junit.xml, so nothing is published and the jenkins/integration-tests check never reports a result at all.

Measured: branch-2026.1 runs 4 xdist workers and finishes the stage in 21.6 min. branch-perf-v17 produces zero [gw worker lines and had only reached 65% of ~181 selected tests when the timeout fired. 83% of its wall time is 18 stalls of ~116-146 s each, one per Docker-backed test.

Root cause: pytest-xdist is only a declared dependency from branch-2026.1 upward. On this branch the string pytest-xdist appears only inside unit_tests/unit/test_keystore.py, never as a dependency, so sct.py integration-tests has no -n option and no way to fan out.

What changed

1. sct.py — added -n (default 4) and pass --dist loadgroup -n{n} to pytest, mirroring upstream/branch-2026.1:sct.py. -p no:warnings is retained (2026.1 dropped it, but changing warning behaviour is out of scope here), and -t/--test is left single-valued to keep the diff focused.

2. xdist_group markers. This branch had zero. Rather than one blanket group, the tests are split into two groups, each a genuine serialization domain, because --dist loadgroup runs different groups on different workers in parallel — so two balanced groups halve the critical path relative to putting every Docker test in one group.

• docker_ssl (18 selected tests) — every module whose tests drive docker_scylla with ssl=True. That path calls create_ca() / create_certificate(), which write fixed shared paths under data_dir/ssl_conf (sdcm/provision/helpers/certificate.py: CA_CERT_FILE, CA_KEY_FILE, CLIENT_CERT_FILE, CLIENT_FACING_CERTFILE, …), and it os.chdir()s the process. Two workers doing this concurrently overwrite each other's certificates, so the client ends up presenting a cert minted for the other worker's container IP. Modules: test_cassandra_stress_thread, test_latte_thread, test_python_driver, test_scylla_bench_thread.

  Note this branch does not have the configure_scylla_node(..., ssl_dir: Path | None = None) per-test SSL directory that branch-2026.1 has, which is exactly why the group is needed here and not there.

• docker_heavy (21 selected tests) — the resource-intensive modules already grouped under this same name on branch-2026.1 (introduced by faf581538b "fix(integration tests): group resource-intensive Docker based tests" — YCSB / vector store / cassandra-stress were failing intermittently in parallel), plus the modules that reuse a fixed docker network name (ycsb_net in test_ycsb_thread, kafka-stack-docker-compose_default in test_kafka). Modules: test_cql_stress_cassandra_stress_thread, test_gemini_thread, test_kafka, test_ndbench_thread, test_run_cqlsh, test_vector_store, test_ycsb_thread.

test_latte_thread and test_scylla_bench_thread mix unit and integration tests in one module, so they get the marker on the individual integration tests rather than via pytestmark — otherwise the marker would leak onto 58 and 21 non-integration tests respectively.

docker_scylla is function-scoped and each test gets its own container on ephemeral host ports (-p <port> with no host port), so Docker-backed tests do not share a container and the remaining 17 ungrouped Docker tests (test_cluster 14, test_alternator_streams_kcl, test_cassandra_harry, test_utils_database_query_utils) parallelise safely alongside the 126 pure-Python config/version tests.

3. pyproject.toml — added pytest-xdist==3.8.0, the same pin as branch-2026.1.

4. docker/env/version — not touched by hand. The New Hydra Version label drives the build_image workflow, which builds and pushes the image and then commits the bump itself (see below).

The hydra image rebuild — done, by the bot

pytest-xdist is NOT present in the currently pinned hydra image. Verified empirically against the exact tag this branch pins (docker/env/version = 1.107-PR13751-82c347a):

$ docker run --rm scylladb/hydra:v1.107-PR13751-82c347a \
    python -c "import xdist; print('xdist', xdist.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import xdist; print('xdist', xdist.__version__)
    ^^^^^^^^^^^^
ModuleNotFoundError: No module named 'xdist'

$ docker run --rm scylladb/hydra:v1.107-PR13751-82c347a bash -lc \
    'which -a python python3 uv; ls -d /*/.venv /opt/*/.venv /home/*/.venv 2>/dev/null'
/usr/local/bin/python
/usr/local/bin/python3
/usr/local/bin/uv
      (no venv anywhere)

$ docker run --rm scylladb/hydra:v1.107-PR13751-82c347a bash -lc \
    'python3 -m pip list | grep -iE "xdist|pytest|execnet"'
pytest                    8.3.4
pytest-random-order       1.0.4
pytest-subtests           0.14.2

$ docker run --rm scylladb/hydra:v1.107-PR13751-82c347a bash -lc \
    'find / -name "xdist*" -maxdepth 8 2>/dev/null | head'
      (no output)

python/python3 are the same /usr/local/bin/python3.13 interpreter (the Dockerfile sets UV_PROJECT_ENVIRONMENT=/usr/local/, so there is no separate venv), and nothing named xdist exists anywhere on the filesystem.

Which Dockerfile builds the image was confirmed before touching anything: docker/env/build_n_push.sh runs uv lock and then docker build -t scylladb/hydra:${VERSION} . from the repo root, so the repo-root Dockerfile is the one (there is no docker/env/Dockerfile on this branch). Its ADD uv.lock . + uv sync --frozen is not inconsistent with uv.lock being absent from git — build_n_push.sh generates the lock immediately before the build, so it is a build artefact rather than a committed file. (#15703 proposes committing it anyway, so --frozen reproduces a pinned resolution instead of one invented seconds earlier — that is independent of this PR and not required by it.) pyproject.toml is therefore the only dependency file to edit, and no uv.lock is committed here.

The image now exists and the bot pinned it: scylla-sct[bot] commit d57bc6ae53 "chore(hydra): create image 1.108-PR15694-82b64e2" set docker/env/version = 1.108-PR15694-82b64e2, and scylladb/hydra:v1.108-PR15694-82b64e2 is published (568 MB, 2026-08-05T18:26:07Z). Confirmed to contain what this PR needs:

$ docker run --rm --entrypoint bash scylladb/hydra:v1.108-PR15694-82b64e2 -c \
    'python3 -V; python3 -c "import xdist,execnet; print(\"xdist\", xdist.__version__, \"| execnet\", execnet.__version__)"; python3 -m pytest --version'
Python 3.13.4
xdist 3.8.0 | execnet 2.1.2
pytest 8.3.4

Why the first attempt produced no image

The version file was originally bumped by hand, in a commit whose subject copied the bot's own format — chore(hydra): create image 1.108-PR15694-d3ec10a. The workflow guards against rebuilding with:

commits_headlines=$(git log origin/master..HEAD --pretty=format:"%s")
if [[ "$commits_headlines" == *"chore(hydra): create image"* ]]; then
    echo "Docker image already built"

so that hand-written subject made the guard match, every build step was skipped, and the run still reported success — while 1.108-PR15694-d3ec10a was never pushed. Every hydra.sh stage then failed on the pull (failed to resolve reference … not found), which is why precommit, unittests, lint_test_cases and integration-tests all reported failure with no test output.

Dropping that commit and re-applying the label was enough. The guard is not otherwise fragile here: actions/checkout uses the default fetch-depth: 1, so origin/master..HEAD only ever sees the tip commit, and the branch's 8 inherited chore(hydra): create image … commits are invisible to it.

Verification

The rebuilt image was built locally from this branch's Dockerfile (not pushed) to prove the dependency actually reaches it:

$ docker build --network=host -t scylladb/hydra:v1.108-local-verify .
   ... => naming to docker.io/scylladb/hydra:v1.108-local-verify   DONE

$ docker run --rm scylladb/hydra:v1.108-local-verify python -c \
    "import xdist, pytest, execnet; print('xdist', xdist.__version__, '| pytest', pytest.__version__, '| execnet', execnet.__version__)"
xdist 3.8.0 | pytest 8.3.4 | execnet 2.1.2

--dist loadgroup pins same-group tests to a single worker, and — importantly — runs the two groups concurrently on different workers:

[gw1] [  5%] PASSED test_grp.py::test_ssl[0]@docker_ssl
[gw1] [ 10%] PASSED test_grp.py::test_ssl[1]@docker_ssl
[gw2] [ 15%] PASSED test_grp.py::test_free[0]
[gw1] [ 20%] PASSED test_grp.py::test_ssl[2]@docker_ssl
...
[gw0] [ 50%] PASSED test_grp.py::test_heavy[0]@docker_heavy
[gw3] [ 55%] PASSED test_grp.py::test_free[1]
[gw0] [ 65%] PASSED test_grp.py::test_heavy[1]@docker_heavy
...
[gw0] [100%] PASSED test_grp.py::test_heavy[5]@docker_heavy
============================== 20 passed in 1.68s ==============================

All 6 docker_ssl tests landed on gw1, all 6 docker_heavy tests on gw0, and the ungrouped ones spread over gw1/gw2/gw3 — note the @docker_ssl / @docker_heavy nodeid suffixes xdist appends for grouped tests.

Group assignment audited across the whole selected suite inside the hydra image (temporary reporting plugin, not committed):

### group=<ungrouped>  tests=142
        1  test_alternator_streams_kcl.py
        1  test_aws_services.py
        4  test_base_version.py
        1  test_cassandra_harry.py
       14  test_cluster.py
       38  test_config.py
       33  test_config_get_version_based_on_conf.py
        1  test_dedicated_hosts.py
        1  test_events.py
        2  test_ssh_none_auth.py
        1  test_utils_database_query_utils.py
       31  test_utils_issues.py
       14  test_version_utils.py
### group=docker_heavy  tests=21
        3  test_cql_stress_cassandra_stress_thread.py
        2  test_gemini_thread.py
        2  test_kafka.py
        4  test_ndbench_thread.py
        3  test_run_cqlsh.py
        3  test_vector_store.py
        4  test_ycsb_thread.py
### group=docker_ssl  tests=18
        9  test_cassandra_stress_thread.py
        4  test_latte_thread.py
        2  test_python_driver.py
        3  test_scylla_bench_thread.py
### total selected = 181

No marker leaks onto a non-integration test, and every module that regenerates the shared certificates is inside docker_ssl.

Collection works with the real argument list:

$ python -m pytest -v -p no:warnings -m integration --dist loadgroup -n4 --collect-only unit_tests/
============= 181/1662 tests collected (1481 deselected) in 8.03s ==============

CLI wiring:

test | opts=['-t', '--test'] | default='' | multiple=False
n | opts=['-n'] | default=4 | multiple=False
junit_xml | opts=['--junit-xml'] | default='' | multiple=False

Pre-commit over the diff — all hooks pass (uv-sort in particular confirms the new dependency is correctly placed):

trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check json...........................................(no files to check)Skipped
detect aws credentials...................................................Passed
detect private key.......................................................Passed
ruff-format..............................................................Passed
ruff.....................................................................Passed
update-conf-docs.....................................(no files to check)Skipped
create-nemesis-yaml..................................(no files to check)Skipped
create-nemesis-pipelines.............................(no files to check)Skipped
uv-sort..................................................................Passed
commitlint...............................................................Passed

This makes the stage finish, not pass

Please do not read still-failing Docker integration tests as "the PR didn't work". There is a separate, independent bug on this branch: unit_tests/conftest.py::configure_scylla_node builds scylla = RemoteDocker(...) and then ends on its two wait.wait_for(...) readiness calls without ever returning it — every return in that function belongs to the nested db_up / db_alternator_up helpers. So fixture_docker_scylla does scylla = configure_scylla_node(...) → yield None → scylla.kill() → AttributeError: 'NoneType' object has no attribute 'kill'.

return scylla is present on master, branch-2026.3, branch-2026.2 and branch-2026.1, and absent on exactly branch-perf-v17 and branch-2025.1 — the two branches whose Docker integration tests mass-ERROR. It is deliberately not fixed here so this PR stays reviewable in isolation (same call as #15691); it is recorded on SCT-714 and will land as its own per-branch change.

So the expected outcome of merging this PR alone is: the stage completes within its timeout and JUnit finally gets published, with the Docker tests still ERRORing — just ~4× faster. It goes green once the return scylla fix lands.

Because the ~2 min per Docker-backed test is genuine container boot rather than an expiring timeout, the critical path is now bounded by the larger group: roughly max(21, 18) × ~2 min. That is what motivated two balanced groups instead of one, and if the stage still runs tight, splitting docker_heavy further (e.g. lifting test_ndbench_thread and test_run_cqlsh, which share no named resource and run ungrouped and green on master) is a one-line change.

unit-tests deliberately left serial

sct.py unit-tests is also serial on this branch and branch-2026.1 runs it with -n2, so parallelising it is desirable. It is not done here: the unit-tests job is currently green, and a local serial-vs-parallel comparison over its ~1481 tests could not be completed (the serial baseline was OOM-killed at 58% in my container), so there is no evidence that parallelising it is safe. Rather than risk a working job on an unverified change, it is left alone and can be done as a follow-up with a proper before/after run.

Out of scope (tracked in SCT-714)

Deliberately not touched: the missing return scylla (above), the container-readiness ERRORs, the missing unit_tests/test_configs/ directory, and the JUnit-on-abort gap.

[fruch]

CI now proves the premise of this PR. Build PR-15694/5 against the freshly published scylladb/hydra:v1.108-PR15694-82b64e2:

============ 22 failed, 157 passed, 2 skipped in 1571.94s (0:26:11) ============
- generated xml file: /home/ubuntu/scylla-cluster-tests/integration-tests-junit.xml -
Recording test results

• The stage finishes and publishes. 26:11 against the 45-min timeout, with all 181 selected tests actually executed. Before this change it was ABORTED at ~65 % of the suite and No test report files were found. Configuration error? — jenkins/integration-tests reported nothing at all.
• The fan-out works. Worker prefixes [gw0]-[gw3] are present where the branch previously produced zero, and the grouped tests carry the @docker_heavy / @docker_ssl nodeid suffix, e.g. [gw0] [ 56%] FAILED unit_tests/test_kafka.py::test_01_kafka_cdc_source_connector@docker_heavy.
• precommit, unittests and lint_test_cases are green.

The 22 remaining failures are not what this PR set out to fix — as stated under "This makes the stage finish, not pass". They are the branch's pre-existing environment and fixture rot, now finally visible instead of hidden behind an abort:

group	example	nature
missing test configs	test_unified_package_aws_sets_ubuntu_user — FileNotFoundError: unit_tests/test_configs/minimal_test_case.yaml	the test_configs/ gap this branch has
live-lookup rot	test_get_branched_repo[...], test_get_specific_tag_of_docker_image[...] — S3 404 / KeyError: docker-image-name; test_images[azure-*] — Azure Image … not found in eastus	same class of rot fixed on master by #15670/#15671
stale signature	test_01_dynamodb_api@docker_heavy — TypeError: YcsbStressThread.__init__() missing 1 required keyword-only argument: cluster_tester	unrelated to parallelism
docker fixtures	test_01_gemini_thread@docker_heavy, test_01_kafka_cdc_source_connector@docker_heavy — UnexpectedExit	already failing when the suite ran serially

One caveat I would rather flag than paper over: test_vector_store.py::test_vector_search@docker_heavy fails with RuntimeError: Vector indexing did not complete within 300 seconds. That module was not among the ones observed failing in the earlier serial runs, so I cannot rule out resource contention from running 4 workers on the same runner. It is grouped, so it is serialised against the other heavy modules, but not against the other three workers. Worth a look under SCT-714 rather than assuming it is pre-existing.

[fruch]

I'll fight the failing integration test in other PRs