fix(integration-tests): run the integration suite in parallel - #15691
fix(integration-tests): run the integration suite in parallel#15691fruch wants to merge 7 commits into
Conversation
a8e725a to
589df36
Compare
589df36 to
da20490
Compare
fruch
added
New Hydra Version
fruch
added
New Hydra Version
8378673 to
d53c7d6
Compare
|
Triaged build PR-15691/4. Good news first: with the image published, the stage now completes — The failures split three ways, and only the first is this PR's fault. 1. The 3
|
d53c7d6 to
2a13de8
Compare
Every test in unit_tests/integration/test_config_get_version_based_on_conf.py
fails before reaching a single assertion:
FileNotFoundError: Couldn't find config file:
/home/ubuntu/scylla-cluster-tests/unit_tests/test_configs/minimal_test_case.yaml
That directory does not exist here. Master moved internal_test_data/ into
unit_tests/test_configs/ in c16e41a ("improvement(treewide): Move
internal_test_data into unit_tests"), which was never backported, so on this
branch the fixtures still live in internal_test_data/. The test file arrived
via cherry-pick carrying master's path.
The two files are byte-identical, so only the path is wrong.
Also corrected the same stale path in test_utils__operator__multitenant_common.py
(latent: setup_default_env sets it, but _multitenant_class_with_shared_options
overwrites SCT_CONFIG_FILES before anything loads it, so nothing failed) and in
the AGENTS.md layout description.
Left unit_tests/test_config_get_version_based_on_conf.py alone on purpose — it
is the stale pre-rename duplicate that #15691 removes, and editing it here
would create a modify/delete conflict.
…test On master, commit d2e3dc7 ("test(unit_tests): move pure integration tests to unit_tests/integration/") renamed this file byte-for-byte to unit_tests/integration/test_config_get_version_based_on_conf.py. That rename was never backported here, so when 55d4096 cherry-picked the follow-up content change it landed as a brand new 352-line file at the integration/ path and left the pre-rename 205-line copy behind. Both copies are marked `pytest.mark.integration` and both are collected. The stale copy still carries the enterprise-prefixed version cases that 55d4096 removed as obsolete (for example `enterprise-2023.1:latest` expecting `is_enterprise=True`), so it can never pass. Every test it defines also exists in the integration/ copy, which is a strict superset, so removing it loses no coverage.
2a13de8 to
43eeadd
Compare
The `integration tests` Jenkins stage on this branch runs single-threaded
and always exceeds its timeout, so the build is ABORTED before pytest can
write its JUnit XML and `jenkins/integration-tests` never reports a
result.
`pytest-xdist` is only a declared dependency from branch-2026.1 upward, so
`sct.py integration-tests` here has no `-n` option and no way to fan the
suite out. Mirror the branch-2026.1 shape: add a `-n` option (default 4)
and pass `--dist loadgroup -n{n}` to pytest, and make `-t/--test`
repeatable so several test files can be selected at once.
`--dist loadgroup` only helps if the container-sharing tests are pinned to
a single worker, so extend the existing `xdist_group("docker_heavy")`
grouping to every integration module that starts a Scylla container via
the `docker_scylla`/`docker_scylla_2` fixture. Those tests cannot be
spread across workers because the fixture reuses fixed docker network
names (`ycsb_net`, `kafka-stack-docker-compose_default`) and, for
`ssl=True`, regenerates the shared certificates under `data_dir/ssl_conf`.
The remaining integration modules are pure-Python config/version tests
and parallelise freely.
Also raise the stage timeout from 40 to 45 minutes so it matches every
other branch.
`pytest-xdist` goes into `requirements.txt` (via `requirements.in`, which
is what `docker/env/Dockerfile` installs from; this branch has no
dependency section in `pyproject.toml` and no `uv.lock`), and
`docker/env/version` is bumped so a fresh hydra image is built and
published -- the currently pinned image does not have xdist installed, and
`-n4` would fail at runtime against a stale tag.
unit_tests/test_microbenchmarking.py covers sdcm/microbenchmarking.py, the old Elasticsearch-backed results analyzer. Master already removed this test in 1747db8 ("refactor(stats): remove Elasticsearch statistics collection and reporting"); nothing else imports the module except docs. It is also the only thing standing between this branch and a green unit-tests stage: all three of its cases assert that the subprocess wrote nothing to stderr, so any import-time warning from a dependency fails them regardless of the code under test. The unit_tests/test_data/test_microbenchmarking/ fixtures are left in place, matching master, which kept them when it dropped the test.
Backport of master's c481dd1 and the arch-aware tag lookup, fixing both ways get_specific_tag_of_docker_image() currently raises: * scylladb/scylla-enterprise-nightly resolved tags under the `enterprise` rolling branch, which stopped producing builds when enterprise development folded into the unified releases. Its relocatables are gone, so the 00-Build.txt fetch 404s. enterprise-2024.1 is the only branch still publishing those images. * 00-Build.txt now carries per-architecture keys, so a bare build_info["docker-image-name"] raises KeyError. Read docker-image-name-<arch> and fall back to the unsuffixed key, and raise a ValueError naming the URL when neither is present rather than a bare KeyError. Fixes both test_get_specific_tag_of_docker_image cases and, through sct_config, test_docker[latest] and test_docker[master:latest].
… branch These eight tests came over with the cherry-pick of the integration test file and exercise sct_config behaviour that only exists on master: * test_relocatable_version_resolves_unified_package (4 params) patches sdcm.sct_config.latest_unified_package, which this branch does not import or define -> AttributeError. * test_unified_package and test_unified_package_aws_sets_ubuntu_user assert use_preinstalled_scylla is forced False -> assert True is False. * test_unified_package_aws_auto_resolves_ami and test_unified_package_aws_verify_passes_without_ami need the AWS auto-resolution of ami_id_db_scylla to the Ubuntu 24.04 base AMI -> "scylla version/repos wasn't configured correctly". This branch has the unified_package option but none of the surrounding logic (relocatable: resolution, forcing use_preinstalled_scylla=False, forcing ami_db_scylla_user=ubuntu, auto-resolving the base AMI), so the tests cannot pass here. Removing them rather than backporting the feature: that is a config-behaviour change to a release branch and belongs in its own PR, not in a test-parallelisation one.
test_13_scylla_version_ami_branch_latest hardcoded branch-5.2:latest. Those AMIs have aged out of eu-west-1, so it fails with "AMIs for scylla_version='branch-5.2:latest' not found". Resolve the branch from the current release instead, the same way master does, using the _get_latest_scylla_release() helper already present in this file. test_sstable_load_utils called cs_thread.parse_results(). That is master's name for it: sdcm/stress/base.py on this branch still calls it verify_results(), so the cherry-picked test raised AttributeError.
43eeadd to
bb4785f
Compare
fruch
added
New Hydra Version
Keeps the integration stage runnable. The label-driven build still does not fire on this branch even after scylladb#15702 landed the paths fix on branch-2025.1: the label was toggled off/on at 2026-08-10T06:59:16Z and 06:59:24Z, this PR changes both requirements.in and requirements.txt, and no build-docker-image run was created (last run on the workflow is from 2026-08-07, on a master-based PR). This image predates the requirements.txt correction, so it still carries the drifted transitive pins (chardet 6.0.0.post1 rather than 5.2.0). Nothing depends on that difference now that the microbenchmarking test is gone, but the image should be rebuilt from the corrected requirements.txt before this merges.
|
Build PR-15691/9:
Every failure fixed here had the same root cause: the integration test files were cherry-picked from master, but the code they exercise was never backported to this branch.
The 5 that remain, and why I left them
Still needs doing before mergeThe The pinned image
|
1 participant
Refs SCT-804, blocks SCT-714.
Problem
The
integration testsstage onbranch-2025.1runs single-threaded and therefore always exceeds its stage timeout. Jenkins ABORTs the build before pytest gets to writeintegration-tests-junit.xml, so nothing is published and thejenkins/integration-testscheck never reports a result at all.Measured:
branch-2026.1runs 4 xdist workers and finishes the stage in 21.6 min.branch-2025.1produces zero[gwworker lines and had only reached 57% of 187 selected tests when the timeout fired — it needs roughly 79 min serially. This branch's timeout is also 40 min, tighter than the 45 min every other branch uses.Root cause:
pytest-xdistis only a declared dependency frombranch-2026.1upward. On this branch the stringpytest-xdistappears only in a comment insideunit_tests/unit/test_keystore.py, never as a dependency, sosct.py integration-testshas no-noption and no way to fan out.What changed
sct.py— added-n(default 4) and pass--dist loadgroup -n{n}to pytest, mirroringupstream/branch-2026.1:sct.py.-t/--testalso becomes repeatable (multiple=True) so several files can be selected at once, matching 2026.1.-p no:warningsis retained (2026.1 dropped it, but changing warning behaviour is out of scope here).xdist_group("docker_heavy")markers — reconciled, not blindly added. This branch already had two occurrences (unit_tests/test_gemini_thread.py,unit_tests/integration/test_sstable_load_utils.py), both already using the namedocker_heavy. Coverage is extended to every integration module that starts a Scylla container via thedocker_scylla/docker_scylla_2fixture, reusing that same single group name — no new or conflicting group names.The rule is deliberately mechanical rather than a per-file judgement call, because these tests genuinely cannot be spread across workers:
unit_tests/conftest.py::configure_scylla_nodereuses fixed docker network names (ycsb_netintest_ycsb_thread/test_alternator_streams_kcl,kafka-stack-docker-compose_defaultintest_kafka) and, forssl=True, regenerates the shared certificates underdata_dir/ssl_confwhileos.chdir-ing the process (test_python_driver,test_latte_thread,test_scylla_bench_thread,test_cassandra_stress_thread).Modules given the marker:
test_kafka,test_latte_thread,test_ndbench_thread,test_cql_stress_cassandra_stress_thread,test_python_driver,test_run_cqlsh,test_cassandra_stress_thread,test_scylla_bench_thread,test_ycsb_thread,test_alternator_streams_kcl,test_cassandra_harry,test_utils_database_query_utils.test_cluster.pyholds mostly non-integration unit tests, so its 4 integration tests get the marker individually rather than viapytestmark.The 6 modules left ungrouped (
test_base_version,test_config,test_events,test_utils_issues,test_version_utils,integration/test_config_get_version_based_on_conf) are pure-Python config/version tests that touch no container and parallelise freely.Jenkinsfile— integration stage timeout 40 → 45 min, matching every other branch.Stale duplicate removed (separate commit) —
unit_tests/test_config_get_version_based_on_conf.py(205 lines) was a leftover duplicate of the currentunit_tests/integration/test_config_get_version_based_on_conf.py(352 lines). Both werepytest.mark.integrationand both were collected. Confirmed obsolete from history:d2e3dc7a99"test(unit_tests): move pure integration tests to unit_tests/integration/" renames this file byte-for-byte (0insertions,0deletions) to theintegration/path.branch-2025.1(git merge-base --is-ancestor→ false), while it is an ancestor ofmaster.55d4096883cherry-picked the follow-up content change it landed as a brand-new 352-line file at theintegration/path and left the pre-rename copy behind.55d4096883explicitly removed as obsolete (e.g.pytest.param("enterprise-2023.1:latest", (None, True))), so it can never pass. Every test it defines also exists in theintegration/copy, which is a strict superset — no coverage is lost.requirements.in/requirements.txt— addedpytest-xdist==3.8.0(same pin asbranch-2026.1).docker/env/Dockerfileinstalls withADD requirements.txt .+uv pip install --system --build-constraint build-constraints.txt -r requirements.txt, sorequirements.txtis the file that reaches the image. Verified this branch'spyproject.tomlcontains only[tool.ruff]config and no dependency section, and there is nouv.lock—requirements.in/requirements.txtis the only dependency path here. Recompiled withuv pip compile requirements.in --generate-hashes --python-version 3.10 --output-file requirements.txt, so the existing file is read as the preference set: the only version lines that change arepytest-xdist==3.8.0and its dependencyexecnet==2.1.2; the rest of the diff is# via ...annotations the previous compile had missed.Blocked on #15702 for the hydra image rebuild
pytest-xdistis NOT present in the currently pinned hydra image. Verified empirically against the exact tag this branch pins (docker/env/version=1.90-2025.1-scylla-driver-v3.29.9):python/python3are the same/usr/local/bininterpreter (no separate venv),pytestis 7.2.0, and nothing namedxdistexists anywhere on the filesystem.docker/env/versionis deliberately left alone — theNew Hydra Versionlabel is supposed to make thebuild_imageworkflow build the image, push it, and commit the bump itself. On this branch it cannot:Until then the stage keeps failing, and an earlier attempt to shortcut it made things worse:
docker/env/versionwas hand-bumped to1.91-2025.1-pytest-xdistwithout anything publishing that tag, so everyhydra.shstage died on the pull (failed to resolve reference "docker.io/scylladb/hydra:v1.91-2025.1-pytest-xdist": not found) andprecommit,unittests,lint_test_casesandintegration-testsall reported failure with no test output. That bump has been reverted.Verification
pytest-xdist==3.8.0is compatible with this branch'spytest==7.2.0:--dist loadgrouppins same-group tests to one worker (smoke test; note the@docker_heavynodeid suffix and that all 6 grouped tests land ongw0):Group assignment audited across the whole selected suite inside the hydra image (temporary reporting plugin, since removed —
SCYLLAmarks tests requestingdocker_scylla/docker_scylla_2):Every module that touches a Scylla container is in the single
docker_heavygroup, and no ungrouped test requests the fixture.Collection works with the real argument list (
141/1313 tests collected; lower than the 187 previously selected because the stale duplicate is gone):Workers actually spin up, and results are unchanged versus serial (the 7 failures are pre-existing and environmental — no GitHub/Jira credentials in my local container — and are identical either way, so parallelism changes no outcome):
CLI wiring:
pytest-xdistinstalls from the regenerated hash-pinnedrequirements.txt:Pre-commit over the diff — all hooks pass:
Out of scope (tracked in SCT-714)
Deliberately not touched: the container-readiness ERRORs (note
unit_tests/conftest.py::configure_scylla_nodeneverreturns itsscyllaobject, so thedocker_scyllafixture yieldsNone), the missingunit_tests/test_configs/directory, and the JUnit-on-abort gap.