Skip to content

Choose ib_neighborhood_radius from the thinnest rank, not the widest - #1866

Open
sbryngelson wants to merge 2 commits into
masterfrom
fix/ib-neighborhood-radius
Open

Choose ib_neighborhood_radius from the thinnest rank, not the widest#1866
sbryngelson wants to merge 2 commits into
masterfrom
fix/ib-neighborhood-radius

Conversation

@sbryngelson

Copy link
Copy Markdown
Member

Written with Claude Code. Addresses the automatic-choice half of #1851.

The bug

ib_neighborhood_radius counts rank hops. A rank keeps an immersed-boundary patch only while the patch centroid lies inside its own subdomain grown outward by that many hops (s_get_neighbor_boundsf_neighborhood_ranks_own_location), and s_handoff_ib_ownership deletes it otherwise. So the radius must span the body in every direction, and the hop that needs the most is the one crossing the thinnest rank.

The automatic choice assembled its rank width the wrong way round:

local_rank_width = -1._wp
do each direction:
    local_rank_width = max(local_rank_width, <this rank's extent in that direction>)
call s_mpi_allreduce_min(local_rank_width, min_rank_width)

Each rank reports its widest extent; the minimum is then taken across ranks. Where every rank is long in one direction and thin in another, the width reported is the long one and the radius comes out too small — the reported "picks 1 and faults".

The fix

Reduce over directions with min rather than max, so the result is the narrowest rank extent over every rank and every direction. Two lines.

How the validation is obtained

examples/3D_ibm_neighborhood_radius/ — a thin plate in a long narrow channel, 20 chords by 8 by 6 with 400×50×50 cells, sized so MFC's topology search settles on 16×2×2 at 64 ranks. That is an ordinary shape for a wake, a jet or a channel, and it makes the ranks strongly anisotropic:

direction ranks extent per rank
x 16 1.250
y 2 4.000
z 2 3.000

The plate is 1.0 × 2.4 × 0.1, so s_get_ib_bound returns 1.3010. Crossing that at 1.250 per hop needs ceil(1.1 × 1.3010 / 1.250) = 2; the old width of 4.000 gives 1.

./mfc.sh run examples/3D_ibm_neighborhood_radius/case.py -n 64
Automatic choice of ib_neighborhood_radius selected:
before 1
after 2

Both runs complete (1 M cells, a few minutes on two CPU nodes). SUMMARY=1 python3 case.py prints the half-extent, the rank extents and the arithmetic without running anything. The README carries the same table and the reasoning.

Two real production grids agree, measured from their own lustre_*_cb.dat:

case topology old width → radius new width → radius
gust encounter, 128 ranks 16×2×4 2.051 → 1 1.052 → 2
flapping wing, 128 ranks 8×4×4 1.745 → 1 1.027 → 2

Scope

The new width is never larger than the old, so the chosen radius never decreases — the change can only make the neighbourhood more conservative, at the cost of more hops in the force reduction. Cases that set ib_neighborhood_radius explicitly are untouched, and so is any near-cubic decomposition, where the widest and narrowest extents coincide. That is why a uniform grid with a balanced topology shows no difference, and why no golden file moves.

This does not fix #1863 (a spurious transverse force on multi-rank IB runs); I checked, and raising the radius there changes nothing.

ib_neighborhood_radius counts rank hops: a rank keeps an immersed-boundary
patch only while the centroid lies inside its subdomain grown by that many
hops, and drops it otherwise. The radius therefore has to span the body in
every direction, and the hop that needs the most is the one crossing the
thinnest rank.

The automatic choice assembled its rank width the wrong way round -- each rank
reported its widest extent and the minimum was taken across ranks. Where every
rank is long in one direction and thin in another, the width reported is the
long one and the radius comes out too small.

examples/3D_ibm_neighborhood_radius is a thin plate in a long narrow channel,
sized so the topology search settles on 16 x 2 x 2 at 64 ranks: x ranks 1.250
wide against y and z at 4.000 and 3.000. The plate's half-extent is 1.3010, so
crossing it takes two hops of 1.250 and one of 4.000. MFC prints radius 1
before this change and 2 after; both runs complete.

The two production grids that motivated it agree: a gust case at 16 x 2 x 4
moves 1 -> 2, and a flapping wing at 8 x 4 x 4 moves 1 -> 2.

The new width is never larger than the old, so the radius never decreases --
the change can only make the neighbourhood more conservative. Cases that set
the parameter explicitly are untouched, as is any near-cubic decomposition,
where the widest and narrowest extents coincide.
Copilot AI lite review requested due to automatic review settings September 12, 2026 18:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Fixes the automatic selection of ib_neighborhood_radius for anisotropic MPI decompositions by basing the hop distance on the thinnest rank extent, and adds a reproducer example documenting/validating the behavior.

Changes:

  • Change rank-width reduction logic from “widest direction per-rank” to “narrowest direction per-rank” when computing the auto ib_neighborhood_radius.
  • Add a new 3D example case that reproduces the issue and prints a quick SUMMARY of the expected arithmetic.
  • Document the reasoning, before/after result, and scope in an accompanying README.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
src/simulation/m_start_up.fpp Fixes the rank-width computation used to auto-select ib_neighborhood_radius on anisotropic decompositions.
examples/3D_ibm_neighborhood_radius/case.py Adds a minimal reproducer case (and optional SUMMARY output) to demonstrate expected radius selection.
examples/3D_ibm_neighborhood_radius/README.md Documents the issue, reasoning, and validation results for the new reproducer case.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +30 to +34
| direction | ranks | extent per rank |
| --- | --- | --- |
| x | 16 | **1.250** |
| y | 2 | 4.000 |
| z | 2 | 3.000 |
Comment on lines +52 to +55
| | printed radius |
| --- | --- |
| before | **1** |
| after | **2** |
Comment on lines +64 to +67
| case | topology | old width | old radius | new width | new radius |
| --- | --- | --- | --- | --- | --- |
| gust encounter, 128 ranks | 16 x 2 x 4 | 2.051 | 1 | 1.052 | **2** |
| flapping wing, 128 ranks | 8 x 4 x 4 | 1.745 | 1 | 1.027 | **2** |
Comment on lines +129 to +130
print(f"rank extents at 64 ranks (16 x 2 x 2): x {(x1 - x0) / 16:.3f}, " f"y {(y1 - y0) / 2:.3f}, z {(z1 - z0) / 2:.3f}")
print(f"hops needed across the thinnest rank: ceil(1.1 * {bound:.4f} / {(x1 - x0) / 16:.3f}) = " f"{max(1, math.ceil(1.1 * bound / ((x1 - x0) / 16)))}")
@codecov

codecov Bot commented Sep 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 61.25%. Comparing base (dc0aec1) to head (15460d3).
⚠️ Report is 4 commits behind head on master.

Files with missing lines Patch % Lines
src/simulation/m_start_up.fpp 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1866      +/-   ##
==========================================
- Coverage   61.26%   61.25%   -0.02%     
==========================================
  Files          84       84              
  Lines       22330    22336       +6     
  Branches     3265     3266       +1     
==========================================
+ Hits        13680    13681       +1     
- Misses       6207     6211       +4     
- Partials     2443     2444       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Immersed-boundary force gains a large spurious transverse component on multiple ranks

2 participants