[Backport 2026.2] fix(upgrade): retry package manager commands during rollback to absorb transient mirror failures

[scylladbbot]

What

Adds retry=3 (and timeout=600 on heavier package-download calls) to the yum remoter calls in UpgradeTest._upgrade_node and UpgradeTest._rollback_node in upgrade_test.py.

Why

Fixes SCT-786: a rolling-upgrade rollback (upgrade_test.UpgradeTest.test_generic_cluster_upgrade) failed when 'sudo yum downgrade scylla* -y' hit a transient Rocky Linux 10 AppStream mirrorlist outage (Cannot prepare internal mirrorlist: No URLs in mirrorlist) while the BaseOS repo, using the same mirrorlist mechanism, succeeded in the same invocation - consistent with a short-lived upstream CDN blip, not an SCT or ScyllaDB defect.

The remoter call for this command had no retry= kwarg, so it defaulted to retry=1, which only retries SSH-transport-level failures, not a non-zero exit from the remote command itself (see sdcm/remote/base.py's RetryMixin._get_retry_params). ~21 equivalent yum call sites in sdcm/cluster.py already pass retry=3/timeout=600 for exactly this class of transient mirror/CDN blip, a documented convention in skills/package-installation/SKILL.md. This PR closes that gap in upgrade_test.py's upgrade/rollback paths.

retry=3 cannot mask a genuinely broken rollback - the retry wrapper re-raises after the final attempt, so a hard failure still fails the test; it only absorbs outages shorter than the retry window (~10s of backoff across 3 attempts) and adds some log noise on a real failure. It does not eliminate SCT-704 recurrence for longer outages.

Scope

• upgrade_test.py: added retry=3/timeout=600 to the yum calls in _upgrade_node (yum update) and _rollback_node (yum remove/install, minor-release downgrade, and the exact SCT-704 failure line - the default 'sudo yum downgrade scylla* -y').
• Deliberately out of scope (per review): the apt-get calls in these same functions, and an unrelated python34-PyYAML yum install elsewhere in the file - kept as hand-written strings / unchanged, since SCT-704 itself is Rocky/yum-only and the apt convention in the codebase is not uniformly settled.
• unit_tests/unit/test_upgrade_rollback_retries.py (new): a direct regression test asserting the exact SCT-704 command now carries retry=3, timeout=600, plus a parametrized sweep over the RHEL/yum rollback-mode branches (major, minor, reinstall) asserting every yum call is retried.

Testing

• python -m pytest unit_tests/unit/test_upgrade_rollback_retries.py -v - 4 passed.
• Verified the reproducer fails on the pre-fix code and passes with the fix.
• python -m pytest unit_tests/unit/ -k 'apt or upgrade' -q - 59 passed (regression sweep).
• ruff/ruff-format and commitlint passed via pre-commit at commit time.

Process

Phases run: plan -> implementer -> reviewer + pragmatist -> implementer (scope-trim revision per review findings). No RCA/reproducer-for-the-flake phase - the underlying event is a non-deterministic upstream mirror outage, not a red/green-reproducible product bug; the fix applies an established, documented repo convention to close a real gap.

Fixes SCT-786

Note

This draft PR was generated by Maia. The root-cause assessment is AI-generated and may be wrong; it requires human review and is never auto-merged.

Testing

• 🟢 rolling-upgrade-rocky10-test #3

• (cherry picked from commit 1b1b530)

• (cherry picked from commit 46e442b)

Parent PR: #15610

[scylladbbot]

@soyacz - This PR has conflicts, therefore it was moved to draft
Please resolve them and mark this PR as ready for review by removing the conflicts label

[scylladb-promoter]

❌ Test Summary: FAILED

❌ Precommit: FAILED

Total	Passed	Failed	Skipped
38	14	2	22

Failed Hooks

ruff-format

Output

- hook id: ruff-format
- exit code: 2

error: Failed to parse upgrade_test.py:343:45: Expected an indented block after `if` statement
345 files left unchanged
345 files left unchanged

ruff

Output

- hook id: ruff
- exit code: 1

invalid-syntax: Expected an indented block after `if` statement
   --> upgrade_test.py:343:45
    |
341 |                     scylla_pkg_ver += f" {scylla_pkg}-machine-image"
342 |             with self.actions_log.action_scope("updating packages"):
343 |                 if node.distro.is_rhel_like:
    |                                             ^
344 | <<<<<<< HEAD
345 |                     node.remoter.run(r"sudo yum update {}\* -y".format(scylla_pkg_ver))
    |

invalid-syntax: Expected a statement
   --> upgrade_test.py:344:1
    |
342 |             with self.actions_log.action_scope("updating packages"):
343 |                 if node.distro.is_rhel_like:
344 | <<<<<<< HEAD
    | ^^
345 |                     node.remoter.run(r"sudo yum update {}\* -y".format(scylla_pkg_ver))
346 | ||||||| parent of 1b1b5303d (fix(upgrade): retry package manager commands in node upgrade and rollback)
    |

invalid-syntax: Expected a statement
   --> upgrade_test.py:344:3
    |
342 |             with self.actions_log.action_scope("updating packages"):
343 |                 if node.distro.is_rhel_like:
344 | <<<<<<< HEAD
    |   ^^
345 |                     node.remoter.run(r"sudo yum update {}\* -y".format(scylla_pkg_ver))
346 | ||||||| parent of 1b1b5303d (fix(upgrade): retry package manager commands in node upgrade and rollback)
    |

invalid-syntax: Expected a statement
   --> upgrade_test.py:344:5
    |
342 |             with self.actions_log.action_scope("updating packages"):
343 |                 if node.distro.is_rhel_like:
344 | <<<<<<< HEAD
    |     ^^
345 |                     node.remoter.run(r"sudo yum update {}\* -y".format(scylla_pkg_ver))
346 | ||||||| parent of 1b1b5303d (fix(upgrade): retry package manager commands in node upgrade and rollback)
    |

invalid-syntax: Expected a statement
   --> upgrade_test.py:344:7
    |
342 |             with self.actions_log.action_scope("updating packages"):
343 |                 if node.d

❌ Tests: FAILED

Total	Passed	Failed	Errors	Skipped
2383	2363	0	1	19

Failed Tests

.unit.test_upgrade_rollback_retries

Type: ERROR
Message: collection failure

Traceback

/usr/local/lib/python3.14/site-packages/_pytest/python.py:507: in importtestmodule
    mod = import_path(
/usr/local/lib/python3.14/site-packages/_pytest/pathlib.py:587: in import_path
    importlib.import_module(module_name)
/usr/local/lib/python3.14/importlib/__init__.py:88: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<frozen importlib._bootstrap>:1398: in _gcd_import
    ???
<frozen importlib._bootstrap>:1371: in _find_and_load
    ???
<frozen importlib._bootstrap>:1342: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:938: in _load_unlocked
    ???
/usr/local/lib/python3.14/site-packages/_pytest/assertion/rewrite.py:197: in exec_module
    exec(co, module.__dict__)
unit_tests/unit/test_upgrade_rollback_retries.py:16: in <module>
    from upgrade_test import UpgradeTest
<frozen importlib._bootstrap>:1371: in _find_and_load
    ???
<frozen importlib._bootstrap>:1342: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:938: in _load_unlocked
    ???
/usr/local/lib/python3.14/site-packages/_pytest/assertion/rewrite.py:188: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/lib/python3.14/site-packages/_pytest/assertion/rewrite.py:357: in _rewrite_test
    tree = ast.parse(source, filename=strfn)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/lib/python3.14/ast.py:46: in parse
    return compile(source, filename, mode, flags,
E     File "/tmp/jenkins/workspace/an_scylla-cluster-tests_PR-15650/upgrade_test.py", line 346
E       ||||||| parent of 1b1b5303d (fix(upgrade): retry package manager commands in node upgrade and rollback)
E                         ^
E   SyntaxError: invalid decimal literal

Full build log