Load-balance, active-box windowing, block-structured AMR

[sbryngelson]

Summary

An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is s_mpi_decompose_computational_domain, refactored through the new m_box module (byte-identical; covered by the existing suite).

• Load-balance infrastructure — m_box (partition arithmetic), m_load_weight/load_weight_wrt (per-cell load-weight field + imbalance metric), m_sfc_partition/sfc_partition_wrt (Morton-SFC predicted-imbalance diagnostic), m_load_balance/load_balance (weighted static decomposition at init; AMR-fine-work-aware), m_rank_timing/rank_time_wrt (per-rank compute-time diagnostic).
• Active-box windowing — m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.
• Hybrid reconstruction/flux sensors — hybrid_weno and hybrid_riemann (+hybrid_weno_eps, hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).
• Block-structured AMR — m_amr + m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.

Full algorithm and user documentation: docs/documentation/amr.md (support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).

AMR physics support matrix (abridged; authoritative table in amr.md)

Supported and golden-tested: single- and multi-fluid (5-eq, mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-block pb/mv side-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) · active_box (blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).

Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) · amr_subcycle under IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).

Validation evidence

• Conservation: reflux-exact (~1e-15 defects) across single/multi-fluid, viscous, bubbles, chemistry, phase change, through subcycled + regridding advances.
• Free-stream preservation: exact (0.0) with subcycling + regrid armed, including 3D, IGR, and Lagrangian configs.
• Decomposition exactness: np=1 == np=2 element-exact with blocks spanning rank seams (fields byte-identical incl. chemistry species).
• Methodology: every feature validated by AMR-vs-no-AMR reference comparison with a control experiment calibrating the expected error scale, plus mechanism evidence from run logs (regrid counts, box coordinates, ghost-point counts) — a golden that does not mechanically exercise its path is treated as no coverage. Liveness goldens for the hybrid sensors are deliberately sized so a silently dead sensor fails them (~5e-4 signal vs 5e-5 tolerance) — they caught a real dead-sensor bug on Cray GPU during CI hardening.
• GPU: full test suite with post-process on 2×V100 — OpenACC 644/644 executed pass, OpenMP offload 643/644 (one slurm launch flake). Speed spot-check (800² 2D blast, 2 ranks): AMR + dynamic regrid costs +3% wall over no-AMR at these sizes; ACC ≈ 12% faster than OMP; acc/omp solutions bitwise identical.
• CI: one fully green 85-lane matrix on record (all GitHub-hosted lanes: gnu/intel × debug/no-debug/reldebug × double/single/mixed, macos, and the NVHPC 23.11–26.3 compat matrix). Fixes landed during hardening were each reproduced locally in the exact failing configuration first (intel FP drift, single-precision tolerance classification, gfortran bounds-check OOB, OpenACC present-table mapping ×3).

Performance

Measured on AMD MI210 (gfx90a), OpenMP offload, one MPI rank per GPU, double precision throughout — MFC and the reference are compared at the same precision. Every figure is a per-step wall differenced between two from-scratch runs of different length, so start-up and mesh ramp-in cancel, and is the mean of three repetitions with the two codes interleaved inside one node reservation.

Reference. AMReX's Tests/GPU/CNS blob test, rebuilt with NUM_GROW = 4 to match MFC's WENO5 ghost width, on a deck matched to MFC's: unit periodic box, entropy blob advected at u=1, gamma 1.4, inviscid, 400^3 base, max_level = 2, ref_ratio = 2, regrid_int = 20, no subcycling, reflux on, density-gradient tagging thresholded to tag the same cells.

AMR overhead on the matched deck (8 GPUs, 400^3 base + 2 levels)

The two codes do not run the same numerics, so the comparable quantity is each code's excess over its own uniform-grid arm at the same advanced-cell count:

excess = AMR s/step  -  uniform s/step x (cells advanced per step / 400^3)

	excess (s/step)	sd
MFC	0.404	0.012-0.035 (two reads, three reps each)
AMReX	0.365	0.013-0.017 (three reads)

MFC pays 1.11x AMReX's AMR overhead on this deck (both codes on the same node, same session; the control build of this branch read 0.557 / 1.56x on the same node the same afternoon, treatment-control-treatment). Two changes carried it there, one commit each:

• Device-resident wire pools. The fine-window exchange (coarse patches to fine blocks, restriction back) staged every box's slice through the host: ~250 per-box map/unmap operations a step, each a synchronous small copy. With rdma_mpi the four wire pools now live on the device and MPI sends and receives them by device address, exactly as the base halo already did; the pack/unpack kernels find the pool present and copy nothing. Pre-registered A/B against the parent build, 3 reps: wall -73 ms/step (t = -5.7), gather phase -47, restriction -35, host unpack/fill -27, transfer waits -20, launches and device work unchanged, goldens byte-identical GPU-vs-GPU with rdma_mpi on.
• The descriptor tax (below).

An earlier revision of this section read 1.73x (0.627 / 0.362) on a different node; MFC's host-bound overhead moves with the host, AMReX's device-bound step does not, so every ratio here is read against a control on the same node. The previous revision of this section read 1.73x (0.627 / 0.362) on a different node; a node-matched control of that revision's code reads 0.615 / 0.384 = 1.60x on this one, so the change described next is worth -0.085 s/step of MFC's excess (-14%), and the rest of the difference is the node (MFC's host-bound overhead moves with the host, AMReX's device-bound step does not). AMR does not slow the base solve: the AMR and uniform arms agree on the base-solve phases to -6.9 ms/step, inside noise.

The descriptor tax (the second change). On amdflang every OpenMP target launch materialises a descriptor for each mapped entity, and a private array costs one per launch (~31 us each); the HLLC kernel privatized 22 of them (47 copies per launch on 137 kernel arguments), WENO5 six, the cons-to-prim kernel six. Declaring those work arrays in a block inside the loop body instead makes them iteration-local by the language and maps nothing: HLLC 47 -> 5 copies per launch, cons-to-prim 16 -> 4, launches unchanged, goldens byte-identical with FMA contraction pinned. Pre-registered A/B against the parent build, 3 reps: wall -69 ms/step (-6.0%, t = -5.2), RHS phase -50 ms/step, fixed cost per batched RHS call 3.7 -> 1.0 ms. (Moving the kernel bodies into device routines takes the copies out just the same and makes the device 35-45% slower on those kernels: the inlined body inherits the routine's register budget. Withdrawn; see gpuParallelization.md.)

Where MFC's 627 ms/step went before that change: 521 ms is AMR machinery, of which 352 ms (56%) is MPI wait. Largest phases: reflux 110, coarse/fine gather 109, restriction 104 ms/step. Reducing that wait is the active work; it is not a correctness or an API concern for this PR.

That wait is imbalance, not a transfer floor. An earlier revision of this section read the per-rank minimum MPI wait (251 ms/step) as a communication floor. That was wrong twice over: the [mpiwait] TOTAL row also sums host-side rows, and a per-rank minimum is not a floor when ranks are waiting on each other. A per-wave-instance trace (every rank's entry and exit for each rendezvous, validated against [mpiwait] to within 1%, and free — an instrumented-vs-parent A/B is a null on every row, largest |t| = 1.9 at n = 3) decomposes it properly: for each instance, the smallest wait any rank achieved is the cost everyone pays, and the rest is one rank waiting on another.

The common cost is 28 ms/step, stable to 3% across three repetitions, and 61-90% of each rank's wait is imbalance. It concentrates in two of the seven wave bands — reflux faces and level-1 restriction carry 78% of it, at 90-96% imbalance — while the parent-fill and restrict-parent bands are floor-dominated. So transfer time and any device synchronisation inside the wait brackets are together bounded by 28 ms/step, and the reachable quantity is load balance.

Scaling

Weak ladder, fixed work per rank, 8 -> 48 GPUs (multi-node), on the lower-order scaling deck with the same geometry and refinement settings:

GPUs	8	16	32	48
s/step	4.040	4.969	6.563	6.785

a fitted 1.222x per rank-doubling, against 1.234x for AMReX on the matched ladder. The scaling slope is at parity — the gap above is a constant factor, not a worsening one.

Notes for anyone reproducing these

• Build with --case-optimization. On this deck it is worth a paired -12.4% on the AMR step (-0.581 s/step, sd 0.084, n=3), 78% of it in the RHS. The excess figures above are case-optimized; without it the same comparison gave 1.80x rather than 1.73x at the time, since case optimization largely cancels in the difference.
• Small cases do not pay. The 800^2 2D spot-check above (+3% wall for AMR + dynamic regrid over no-AMR) is the other end of the range: too little fine work to amortize the machinery. This is expected, and is why every flag ships default-off.
• Third digit is noise. The MFC/AMReX excess ratio drifts about 4% run-to-run on this machine (the non-case-optimized ratio measured 1.73x, 1.75x, 1.80x and 1.88x for the same comparison in four sessions). The master merge in this branch did not regress it: 1.87x merged vs 1.88x pre-merge, paired excess delta -0.004 +/- 0.013.

Known issues (all non-gating or in progress)

• Frontier/Cray, in progress: two device-mapping issues in the IB machinery under CCE (plain-IBM descriptor corruption on gpu-omp; a present-table crash in the AMR-IB swap on gpu-acc). Being fixed with direct Cray iteration; the CCE bench/case-opt lane failures share the same root.
• NVHPC 24.1/24.3 compat lanes (informational, continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK, -tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.
• Accuracy postures documented rather than hidden: multi-fluid alpha-sum deviation bounded ~5.7e-3 at historical block faces during shock crossing (non-growing); viscous prolongation-ghost seam ~1e-6 np-dependence; IGR seam conservation is truncation-order (no reflux capture from the fused IGR kernels — exact-conservation capture is noted future work).

Review guide

The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).

All parameters ship default-off with case_validator entries, runtime checker gates, and case.md/amr.md documentation.

[Copilot]

Pull request overview

This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.

Changes:

• Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
• Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
• Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.

Reviewed changes

Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
toolchain/mfc/params/descriptions.py	Adds user-facing descriptions for new experimental/performance parameters.
toolchain/mfc/params/definitions.py	Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability.
toolchain/mfc/lint_docs.py	Treats new validator checks as non-physics doc checks.
tests/F980C769/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/ECABA006/golden-metadata.txt	Adds golden metadata for active-box test coverage.
tests/DD4CD8F3/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/CC4213FD/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/BD21A5C0/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/BCBA6E74/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/ACE05393/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/987D9025/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/852CCB81/golden-metadata.txt	Adds golden metadata for AMR-related golden tests.
tests/65C375B4/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/4DADE04B/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/454C565F/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/3A474BEE/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/2FC423D3/golden-metadata.txt	Adds golden metadata for a new/updated test.
tests/13945217/golden-metadata.txt	Adds golden metadata for moving-IB under AMR test coverage.
src/simulation/m_viscous.fpp	Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation.
src/simulation/m_time_steppers.fpp	Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations.
src/simulation/m_start_up.fpp	Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O.
src/simulation/m_sfc_partition.fpp	Adds analysis-only SFC tiling + weighted partition prediction and reporting.
src/simulation/m_rank_timing.fpp	Adds per-rank wall-time imbalance measurement helpers and reporting.
src/simulation/m_load_weight.fpp	Adds per-cell load-weight field construction and rank-level imbalance reporting.
src/simulation/m_hypoelastic.fpp	Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps).
src/simulation/m_global_parameters.fpp	Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters.
src/simulation/m_data_output.fpp	Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics.
src/simulation/m_checker.fpp	Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations.
src/simulation/m_active_box.fpp	Adds active-box initialization/growth and debug envelope checking.
src/simulation/m_acoustic_src.fpp	Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort).
src/post_process/m_start_up.fpp	Calls AMR fine-data reader and AMR overlay writer when amr is enabled.
src/post_process/m_global_parameters.fpp	Adds default-off amr flag for post_process overlay behavior.
src/post_process/m_data_output.fpp	Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration.
src/common/m_phase_change.fpp	Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting.
src/common/m_global_parameters_common.fpp	Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros.
src/common/m_derived_types.fpp	Introduces a simple t_box type used by new partitioning infrastructure.
src/common/m_box.fpp	Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits).
src/common/m_boundary_common.fpp	Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill.
docs/module_categories.json	Registers new modules under documentation categories.
docs/documentation/readme.md	Adds AMR section link to the documentation index.
.typos.toml	Adds project-specific abbreviations to the spelling allowlist.

[codecov]

Codecov Report

❌ Patch coverage is 57.28000% with 1068 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.60%. Comparing base (dc0aec1) to head (e9e4c0b).
⚠️ Report is 9 commits behind head on master.

Files with missing lines	Patch %	Lines
src/simulation/m_amr_registers.fpp	62.19%	91 Missing and 143 partials ⚠️
src/post_process/m_data_input.f90	38.83%	94 Missing and 32 partials ⚠️
src/simulation/m_riemann_solver_hlld.fpp	2.75%	105 Missing and 1 partial ⚠️
src/simulation/m_amr_restart.fpp	54.14%	67 Missing and 38 partials ⚠️
src/simulation/m_riemann_solver_hll.fpp	70.58%	45 Missing and 35 partials ⚠️
src/simulation/m_rhs.fpp	64.73%	46 Missing and 15 partials ⚠️
src/simulation/m_load_balance.fpp	46.80%	42 Missing and 8 partials ⚠️
src/post_process/m_data_output.fpp	67.27%	28 Missing and 8 partials ⚠️
src/simulation/m_ibm.fpp	72.30%	19 Missing and 17 partials ⚠️
src/simulation/m_amr_xchg_audit.fpp	56.00%	29 Missing and 4 partials ⚠️
... and 21 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1628      +/-   ##
==========================================
- Coverage   61.26%   60.60%   -0.67%     
==========================================
  Files          84       96      +12     
  Lines       22330    31022    +8692     
  Branches     3265     5040    +1775     
==========================================
+ Hits        13680    18800    +5120     
- Misses       6207     8423    +2216     
- Partials     2443     3799    +1356     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sbryngelson]

Upstream latent gap found during the MHD+AMR investigation (independent of this PR): m_riemann_solver_hlld.fpp contains no GLM terms, so hyper_cleaning = T with riemann_solver = 4 (HLLD) is silently inert — psi never evolves and no cleaning occurs. The existing hyper_cleaning golden uses HLL, which does carry the GLM coupling, so CI never sees this. Worth either adding GLM terms to HLLD or a validator prohibit on the combination.

Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1).

[sbryngelson]

active_box is single-rank-only — make it MPI-real, or defer it?

Raising this as a self-review point. active_box disables itself with a warning at num_procs > 1 and falls back to full-domain compute. Since production runs here are essentially always multi-rank, that makes the optimization a no-op (plus a warning line and some maintenance surface) in exactly the configurations that matter. We should decide what to do with it.

Why it's single-rank today. The active box is a single global, coarse-cell-indexed region. On one rank you just clip the RHS loops (convert / reconstruct / Riemann) to it. Under domain decomposition the geometry is easy — each rank intersects the global box with its subdomain and freezes the exterior — but it creates a load-imbalance problem: if the disturbance occupies only a few ranks' territory, those ranks do all the work while the rest idle, and the slowest rank sets wall-clock, so you save nothing. The current code sidesteps this by disabling rather than solving it.

The enabling pieces are already in this PR. A multi-rank active box needs (a) per-rank clipping of the box (straightforward), and (b) a way to keep otherwise-idle ranks busy — i.e. redistribute the domain so the active region spreads across ranks. That second part is exactly what this PR's load-balance arc (m_load_balance / m_sfc_partition / m_load_weight) provides, and the active box already produces the natural work-weight signal (active vs. frozen cells) that a rebalancer would consume. The two features are built to compose — active box marks where the work is, load-balance moves ranks to it — they're just not wired together yet.

Options:

1. Make it MPI-real. Per-rank box clipping + feed the active/frozen cell count into the load-weight so load_balance redistributes toward the active region. This is the version that actually helps at scale, and it's the natural payoff of shipping the load-balance infrastructure in the same PR.
2. Defer / gate it. Drop active_box from this PR (or keep it behind an explicit experimental flag with the single-rank limitation documented up front) and keep the performance family focused on AMR, which is already multi-rank-capable via the P2P device-path gather/scatter. Re-introduce active box once the load-balance integration is designed.

Leaning toward (1) if there's appetite, since the enabling infra is right here; otherwise (2) keeps the PR's surface honest — a default-off single-node optimization advertised in an exascale code invites confusion. At minimum, the docs should state plainly that it is single-rank-only today.

[github-actions]

Lines of Code

File	Lines	Diff
src/simulation/m_amr.fpp	8595	+8595
src/simulation/m_amr_regrid.fpp	2160	+2160
src/simulation/m_amr_registers.fpp	1048	+1048
src/simulation/m_amr_restart.fpp	383	+383
src/post_process/m_data_input.f90	745	+314
src/simulation/m_phase_timing.fpp	274	+274
src/simulation/m_ibm.fpp	1474	+221
src/simulation/m_amr_xchg_audit.fpp	210	+210
src/simulation/m_time_steppers.fpp	1042	+181
src/simulation/m_sfc_partition.fpp	169	+169
src/simulation/m_load_balance.fpp	164	+164
src/post_process/m_data_output.fpp	1338	+151
src/simulation/m_load_weight.fpp	121	+121
src/simulation/m_active_box.fpp	118	+118
src/simulation/m_rhs.fpp	2056	+100
src/simulation/m_global_parameters.fpp	867	+90
src/common/m_box.fpp	71	+71
src/simulation/m_riemann_solver_hllc.fpp	1388	+69
src/simulation/m_start_up.fpp	1320	+59
src/simulation/m_rank_timing.fpp	50	+50
src/common/m_mpi_common.fpp	1532	+48
src/simulation/m_data_output.fpp	1354	+48
src/simulation/m_weno.fpp	1380	+40
src/simulation/m_acoustic_src.fpp	551	+36
src/simulation/m_riemann_solver_lf.fpp	548	+31
src/simulation/m_riemann_solvers.fpp	137	+25
src/common/m_phase_change.fpp	306	+22
src/common/include/2dHardcodedIC.fpp	453	+21
src/simulation/m_bubbles_EL.fpp	1648	+17
src/common/include/omp_macros.fpp	372	+14
src/simulation/m_checker.fpp	84	+14
src/common/include/macros.fpp	141	+13
src/post_process/p_main.fpp	61	+7
src/simulation/m_igr.fpp	2295	+7
src/common/m_constants.fpp	98	+6
src/simulation/m_hypoelastic.fpp	640	+6
src/simulation/m_riemann_solver_hlld.fpp	198	+6
src/simulation/m_riemann_state.fpp	1187	-6
src/common/m_boundary_common.fpp	453	+5
src/common/m_variables_conversion.fpp	1470	+5
src/common/m_derived_types.fpp	474	+4
src/common/include/3dHardcodedIC.fpp	250	+3
src/common/m_boundary_io.fpp	271	-3
src/simulation/m_riemann_solver_hll.fpp	618	+3
src/simulation/m_surface_tension.fpp	266	+3
src/common/m_global_parameters_common.fpp	244	-2
src/post_process/m_start_up.fpp	788	+2
src/simulation/m_cbc.fpp	1116	-2
src/post_process/m_global_parameters.fpp	400	+1
src/pre_process/m_data_output.fpp	625	+1
src/simulation/m_bubbles_EE.fpp	304	+1
src/simulation/m_thinc.fpp	313	+1

Directory	Lines	Diff
common	10581	+207
pre_process	4514	+1
simulation	42146	+14242
post_process	3874	+475
total	61115	+14925