Probe the node with a solver binary, not just syscheck

[sbryngelson]

#1865 excluded atl1-1-01-002-28-0 by hand. This closes the gap that made the hand-edit necessary, so the next such node is caught by the machinery that already exists.

Why the existing probe missed it

node-exclude.sh says the preflight is meant to catch "SIGILL from a binary built for another microarchitecture", and run_case_optimization.sh notes Phoenix's compute nodes are heterogeneous. The wiring is all there - the case-opt lane calls preflight.sh, which emits MFC_FAULT_NODE for submit-slurm-job.sh to read back and exclude.

It never fired, because the probe is syscheck. syscheck initialises MPI, makes a device context and launches a trivial kernel - a few hundred lines that do not contain the wide vector instructions the solver does. A binary built on one microarchitecture and run on an older one passes it and then dies in the real work.

Measured over 2026-09-11..12: atl1-1-01-002-28-0 SIGILLed in pre_process on all five case-optimization benchmarks, across four attempts spanning 24 hours, at s_assign_default_values_to_user_inputs. syscheck passed on that node every time. No passing Case Opt job on any PR ran there; the same node also produced tolerance failures on #1843's test suite and the "non-physical phasic density" aborts quoted in #1865.

Local work that pinned it down, for the record: the change #1845 was blamed for is provably inert (the toolchain's generated Fortran is byte-identical at num_probes_max 10 and 64, and the constant appears nowhere outside src/simulation/), and CI's exact build - same slug gpu-mp-0e981924c0, case-optimized, --gpu mp, MPI - runs clean under both NVHPC 24.1 and 25.5 here. What #1845 actually did was change m_constants.fpp and so force a full rebuild, which is what put a freshly-built binary on a node it had not been built for.

What this does

After syscheck passes, probe again with the newest pre_process under build/install. It is compiled from the same vectorised Fortran the tests run, and it reaches s_assign_default_values_to_user_inputs - exactly where the observed SIGILL landed - before it needs an input file, so it faults on a mismatched node without a case directory.

Only SIGILL counts. Run without a case, pre_process fails for a dozen ordinary reasons - no input file, no restart data, a missing module - and none of them say anything about the node. Anything that is not signal 4 is logged and ignored:

Preflight: build/install/.../pre_process started here (status 1, not SIGILL); node accepted.

That asymmetry is deliberate and matches the rest of the script, which already refuses to judge a node on a launch that never happened: excluding a healthy node costs three allocations and is far worse than missing one bad one.

If no pre_process is built yet, the probe returns without an opinion, so nothing changes for callers that have not built one.

Testing

Exercised the function directly against two stubs, since reproducing a real microarchitecture mismatch needs two unlike nodes:

• a binary that raises SIGILL - emits MFC_FAULT_NODE=<node> and exits 77, so the wrapper excludes and resubmits.
• a binary that exits 1 with "input file not found" - logged and accepted, node not excluded.

bash -n clean, ./mfc.sh precheck passes 7/7.

The real check is the next time a node mismatch happens: it should exclude itself and resubmit instead of failing five benchmarks four times over. Worth keeping #1865's static entry either way - this is the safety net, not a replacement.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Updates the CI preflight to detect CPU microarchitecture mismatches by probing the node with a real solver binary (pre_process) after syscheck passes, so SIGILL failures can be auto-excluded instead of requiring manual node bans.

Changes:

• Add an isa_probe that runs the newest build/install/**/pre_process and excludes the node only on SIGILL.
• Invoke isa_probe after a successful syscheck probe.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Claude Code Review

Head SHA: 63483e8

Files changed:

• 1
• .github/scripts/preflight.sh

Findings:

• isa_probe() locates the probe binary with find build/install -name pre_process -type f -printf '%T@ %p\n' ... 2>/dev/null, but -printf is GNU-only. This is the exact failure mode the adjacent newest_syscheck() (a few lines above, same file) documents and avoids by using ls -t instead: on a BSD/macOS find, -printf errors out, is swallowed by 2>/dev/null, and isa_bin comes back empty every time. Since the code treats an empty isa_bin as "nothing to probe with" ([ -n "$isa_bin" ] || return 0), the new SIGILL check silently never runs on those platforms instead of failing loudly.
• isa_probe() picks the globally newest pre_process under build/install with no device filter, whereas newest_syscheck() first restricts the search to -path "*${device}*" before falling back. If both a cpu and a gpu-* install exist on the node (a scenario the surrounding code explicitly guards against for syscheck), isa_probe can end up testing a pre_process binary built for a different device than the one this allocation is validating, undermining the stated rationale that the probe binary is "compiled from the same vectorised Fortran the tests run."

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.25%. Comparing base (dc0aec1) to head (0e367c7).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1871      +/-   ##
==========================================
- Coverage   61.26%   61.25%   -0.02%     
==========================================
  Files          84       84              
  Lines       22330    22336       +6     
  Branches     3265     3266       +1     
==========================================
+ Hits        13680    13681       +1     
- Misses       6207     6211       +4     
- Partials     2443     2444       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.