fix: deflake //rs/tests/consensus/upgrade:upgrade_downgrade_unassigned_nodes_test - #11412
Merged
pierugo-dfinity merged 2 commits intoSep 4, 2026
Conversation
…d_nodes_test The test performs two sequential GuestOS upgrades of the unassigned node, each polled with a 600s retry budget, but sets no driver timeouts, so the SystemTestGroup default per-test timeout of 10 minutes applies: a single exhausted poll alone eats the whole budget by design. The bazel test_timeout also defaulted to "long" (900s), leaving ~700s for setup + test + teardown. Give the test explicit driver timeouts covering its own worst case (25min per test / 30min overall) and set test_timeout = "eternal" on both targets sharing the binary, like the upgrade_downgrade_*_subnet_test siblings in this package. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The timeout hierarchy is consistent with sibling upgrade tests and safely covers both polling budgets.
Pull request overview
Extends timeout budgets to prevent legitimate dual GuestOS upgrades from exceeding test limits.
Changes:
- Adds 30-minute overall and 25-minute per-test driver timeouts.
- Sets related Bazel targets to the
eternaltimeout class.
File summaries
| File | Description |
|---|---|
rs/tests/consensus/upgrade/upgrade_downgrade_unassigned_nodes_test.rs |
Configures explicit driver timeouts. |
rs/tests/consensus/upgrade/BUILD.bazel |
Extends Bazel timeouts for both variants. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
pull Bot
pushed a commit
to mikeyhodl/ic
that referenced
this pull request
Sep 2, 2026
…-hook socket (dfinity#11413) # Why While root-causing a flake of `//rs/tests/consensus/upgrade:upgrade_downgrade_unassigned_nodes_test_head_nns` (2026-08-28 17:40 UTC), the unassigned GuestOS node turned out to have **no working DNS for its entire ~11-minute life**: * 443 × enforced SELinux denials in its journal: ``` avc: denied { connectto } for comm="systemd-resolve" path="/run/systemd/resolve.hook/io.systemd.Network" scontext=...:systemd_resolved_t:... tcontext=...:systemd_networkd_t:... tclass=unix_stream_socket permissive=0 ``` * 790 × `Failed to start query: Permission denied` from systemd-resolved — every DNS lookup failed; * consequently, all 58 of the orchestrator's attempts to download the elected GuestOS upgrade image failed within ~2 ms (`error sending request`), zero bytes ever fetched, and the test timed out. systemd-resolved connects to systemd-networkd's varlink "resolve hook" socket (`/run/systemd/resolve.hook/io.systemd.Network`) to obtain per-link DNS configuration. Our policy does not allow that `connectto`, so the denial fires on **every** boot (a single `Failed to connect to resolve.hook: Permission denied` appears in healthy boots too). Usually resolved still ends up with working DNS through other paths; nondeterministically — as in this run — it does not recover and the node is left without name resolution, with no self-healing for the rest of the boot. SSH/IP-based connectivity keeps working, which is why the node looks healthy to the test driver while every DNS-dependent operation (such as upgrade-image downloads) is dead. ## Fix Allow the connection in the `systemd-fixes` policy module: ``` allow systemd_resolved_t systemd_networkd_t : unix_stream_socket { connectto }; ``` (The sock-file write permission is evidently already granted — the denial fires on the final `connectto` check.) ## Validation * `bazel build //ic-os/guestos/envs/dev:rootfs-tree.tar` — pass (this is the action that compiles the SELinux policy modules via the refpolicy devel Makefile, so it validates the new rule); `//ic-os/components:check_unused_components_test` — pass. (`dev_component_file_references_test` fails identically on unmodified master on this machine — a local `UnicodeDecodeError` reading the `open_rootfs_dev` binary — so it is left to CI.) Found while root-causing the `//rs/tests/consensus/upgrade:...` flakiness following `.claude/skills/fix-flaky-tests/SKILL.md`. The tightened driver/bazel timeouts for the affected test are in dfinity#11412. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
basvandijk
marked this pull request as ready for review
September 3, 2026 07:16
basvandijk
enabled auto-merge
September 3, 2026 07:16
pierugo-dfinity
approved these changes
Sep 3, 2026
pierugo-dfinity
left a comment
pierugo-dfinity
left a comment
Contributor
There was a problem hiding this comment.
Thanks Bas for looking into it. Maybe we could tell Claude to yap less in PR descriptions. To me, just saying
This aligns the test with its siblings in this package (upgrade_downgrade_{app,nns}_subnet_test set 30–35 min driver timeouts and test_timeout = "eternal")
was enough of a justification, the precise timeout numbers don't matter much. The Note on an independent bug and the whole Validation paragraph could also be trimmed out.
basvandijk
disabled auto-merge
September 3, 2026 07:36
pierugo-dfinity
enabled auto-merge
September 4, 2026 09:50
pierugo-dfinity
deleted the
ai/deflake-upgrade_downgrade_unassigned_nodes_test-2026-09-01
branch
September 4, 2026 10:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR slightly rewrites the upgrade system tests code to reuse duplicated constants and sets the same timeout to all of them. In particular,
upgrade_downgrade_unassigned_nodes_testnow has the same timeout as its siblingsupgrade_downgrade_{app,nns}_subnet_test. This deflakes the former as it was occasionally timing out.🤖 Generated with Claude Code
👨 Revised by @pierugo-dfinity