Summary
Repeated, identical tracking/solve requests return run-to-run non-identical results when OpenMP threading is enabled (OMP_NUM_THREADS > 1), and bit-identical results when it is disabled (OMP_NUM_THREADS=1). The differences are last-bit at first, but because the adaptive-precision (AMP) tracker makes precision/step decisions off norms and condition estimates, a last-bit difference occasionally tips a decision and the path diverges — so the final endpoint can differ, and downstream algorithms (here, a numerical cellular decomposition) get materially different output (e.g. a surface decomposition that is clean in one run and has ~18 spurious cell-construction problems in another, same system, same seed).
Evidence
Driving the same seeded surface decomposition twice through the Python bindings:
OMP_NUM_THREADS=1 (serial or fork-pool): two runs produced bit-identical geometry (SHA-256 over sorted, rounded vertex coordinates matched exactly: 7443b55d...).OMP_NUM_THREADS unset / >1: the same two runs produced different geometry and different validation-problem counts.
The fork-based process parallelism in the caller is deterministic (results merged in a fixed task order); the non-determinism is entirely inside the (OpenMP-threaded) numeric core.
Likely root cause
Non-associative floating-point reductions parallelized with OpenMP (dot products, vector norms, matrix-vector accumulations in the linear-algebra / predictor-corrector paths) sum their partial results in thread-scheduling order, which varies run to run. a + b + c in FP is not associative, so the last bits vary; AMP then amplifies an otherwise-negligible last-bit wobble into an occasional path divergence.
Impact
- Reproducibility: identical asks are not reproducible under threading — bad for debugging, regression tests, pinned fixtures, and provenance/records that claim a system+seed determines an answer.
- Forces
OMP_NUM_THREADS=1 for reproducible runs, which single-threads the serial solve phases and gives up that parallelism.
Proposed fix
Make the core reductions deterministic regardless of thread count — e.g. fixed-order (tree) reductions, per-thread partials combined in a canonical order, or a deterministic/reproducible mode flag. The goal (user's words): deterministic results regardless of parallel level / threading, not merely at OMP_NUM_THREADS=1.
Notes
Observed via the pure-Python cellular-decomposition work driving many repeated solves; happy to provide a minimal C++ repro (repeated identical Track/Solve at OMP_NUM_THREADS>1, hashing endpoints) if useful.
Summary
Repeated, identical tracking/solve requests return run-to-run non-identical results when OpenMP threading is enabled (
OMP_NUM_THREADS > 1), and bit-identical results when it is disabled (OMP_NUM_THREADS=1). The differences are last-bit at first, but because the adaptive-precision (AMP) tracker makes precision/step decisions off norms and condition estimates, a last-bit difference occasionally tips a decision and the path diverges — so the final endpoint can differ, and downstream algorithms (here, a numerical cellular decomposition) get materially different output (e.g. a surface decomposition that is clean in one run and has ~18 spurious cell-construction problems in another, same system, same seed).Evidence
Driving the same seeded surface decomposition twice through the Python bindings:
OMP_NUM_THREADS=1(serial or fork-pool): two runs produced bit-identical geometry (SHA-256 over sorted, rounded vertex coordinates matched exactly:7443b55d...).OMP_NUM_THREADSunset />1: the same two runs produced different geometry and different validation-problem counts.The fork-based process parallelism in the caller is deterministic (results merged in a fixed task order); the non-determinism is entirely inside the (OpenMP-threaded) numeric core.
Likely root cause
Non-associative floating-point reductions parallelized with OpenMP (dot products, vector norms, matrix-vector accumulations in the linear-algebra / predictor-corrector paths) sum their partial results in thread-scheduling order, which varies run to run.
a + b + cin FP is not associative, so the last bits vary; AMP then amplifies an otherwise-negligible last-bit wobble into an occasional path divergence.Impact
OMP_NUM_THREADS=1for reproducible runs, which single-threads the serial solve phases and gives up that parallelism.Proposed fix
Make the core reductions deterministic regardless of thread count — e.g. fixed-order (tree) reductions, per-thread partials combined in a canonical order, or a
deterministic/reproduciblemode flag. The goal (user's words): deterministic results regardless of parallel level / threading, not merely atOMP_NUM_THREADS=1.Notes
Observed via the pure-Python cellular-decomposition work driving many repeated solves; happy to provide a minimal C++ repro (repeated identical
Track/SolveatOMP_NUM_THREADS>1, hashing endpoints) if useful.