fix: don't fail system-test runs when the Vector logging VM is unreachable - #11411
Draft
basvandijk wants to merge 1 commit into
Draft
fix: don't fail system-test runs when the Vector logging VM is unreachable#11411basvandijk wants to merge 1 commit into
basvandijk wants to merge 1 commit into
Conversation
…hable The vector_logging task supervises the whole test plan, so a panic in it SIGKILLs the actual test and fails the run, even though the Vector VM only ships logs to Elasticsearch and has no interaction with the system under test. This happened in the flaky run of upgrade_downgrade_nns_subnet_test_head_nns_colocate on 2026-09-01: the Vector VM was allocated by Farm but never became reachable, and VectorVm::sync_with_vector panicked after 503s of SSH retries, killing an otherwise healthy test mid-upgrade. Propagate errors on the SSH/remote-command paths of sync_with_vector (the calling loop already warns and retries every 30s), and warn-and-return when VectorVm::start fails (a supervisor task finishing normally has no effect on its supervised children, unlike a failure). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Remote container retries can become permanently stuck, and warnings omit underlying error details.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Prevents optional Vector logging failures from aborting system tests.
Changes:
- Propagates Vector SSH and remote-command errors.
- Continues tests when Vector VM provisioning fails.
File summaries
| File | Description |
|---|---|
vector_vm.rs |
Replaces panic paths with recoverable errors. |
vector_logging_task.rs |
Warns and exits cleanly when startup fails. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ), | ||
| ) | ||
| .unwrap(); | ||
| .context("Failed to run vector container")?; |
| let session = deployed_vm | ||
| .block_on_ssh_session() | ||
| .unwrap_or_else(|e| panic!("Failed to setup SSH session to vector because: {e:?}!",)); | ||
| .context("Failed to setup SSH session to vector")?; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
In the flaky run of
//rs/tests/consensus/upgrade:upgrade_downgrade_nns_subnet_test_head_nns_colocateon 2026-09-01 18:13 UTC, the only failing task wasvector_logging:The Vector VM (a universal VM that only ships logs to Elasticsearch/Kibana — pure observability, no interaction with the system under test) was allocated by Farm but never became reachable over SSH.
VectorVm::sync_with_vectorthen panicked (rs/tests/driver/src/driver/vector_vm.rs:271), and since thevector_loggingtask is composed as a supervisor of the whole test plan (rs/tests/driver/src/driver/group.rs), the panic SIGKILLed the actual test mid-upgrade and failed the run — even though the IC under test was perfectly healthy.Fix
Turn the panics/unwraps on the SSH/remote-command paths of
sync_with_vectorinto propagated errors. The calling loop invector_logging_taskalready warns and retries every 30 s onErr(that behavior was introduced for the sync path in ce616a3), andconfig_hashis only updated after a fully successful sync, so a failed sync is naturally retried with the same config.Also soften the remaining panic path,
VectorVm::start(...).expect(...), to warn-and-return: per the task-scheduler semantics (action_graph.rs), a supervisor task finishing normally has no effect on its supervised children — only a failure cancels them — so returning after a warn simply gives up vector logging for that run and lets the tests proceed. No retry ofstart()is attempted: Farm's HTTP layer already retries each call for up to 500 s internally, and re-running a partially-failedstart()would re-issuecreateVmwith the same VM name, which is not known to be idempotent.No behavior change when the Vector VM works: all edits are on error paths only.
Trade-off: a genuine regression in Vector VM provisioning now surfaces as grep-able
Failed to start Vector VM/Failed to sync with vector vmwarnings instead of failing runs — consistent with how the other observability tasks (logs_stream,log_consoles,metrics_sync) already behave.Validation
cargo check --all-targets --all-features -p ic-system-test-driver,cargo fmt,./ci/scripts/rust-lint.sh— clean//rs/tests/consensus/upgrade:upgrade_downgrade_unassigned_nodes_test_head_nnsPASSED in 177s on this branch, with "Spawned vector vm" and "Vector targets sync complete" present in the driver log (unchanged happy path);//rs/tests/driver:unit_testsand//rs/tests/idx:test_e2e_scenariospassFound while root-causing the
//rs/tests/consensus/upgrade:...flakiness following.claude/skills/fix-flaky-tests/SKILL.md.🤖 Generated with Claude Code