Your question
The qualification and final rounds have each team run on their own machine, with
seeds drawn on the day. That makes cross-machine numeric variation a property of
the format rather than a detail of anyone's setup, and I could not find anywhere
in the repo that says how much of it to expect.
We measured it because we needed the answer for our own workflow, and the number
turned out to be small enough that it may be worth stating in the docs. Two
questions:
- Does the organiser treat cross-machine variation as a factor in the live
rounds, or is a run on a competitor's machine taken as authoritative?
- Would a line in CONTRIBUTING.md recording a measured bound be welcome? The
baseline section already explains BLAS jitter in the stored baselines, and
this is the same phenomenon reaching the score instead of the fixtures.
Context
Two machines, scenario 1, seed 0. The only environment difference we could find
was a Python patch release, 3.14.6 against 3.14.7. The full-precision digests of
the rocket trajectory differed; the balloon field matched.
A differing digest is not by itself a finding, since a difference of 1e-13 m and
a difference of 100 m produce the same one. So we measured against a
perturbation larger than a patch release instead, scipy 1.16.2 against 1.17.1:
balloon field 2.9 mm worst case, median 0, launch positions identical
trajectory 2e-8 m at the end of the flight
Seven orders of magnitude below anything that moves a 1.5 m pop verdict. On that
evidence we treat runs from different machines as comparable and rank by a
multi-seed median rather than by digest equality.
The sensitivity ran opposite to our guess, which is the part we would not have
predicted from the code: a scipy bump moves the balloon field and leaves the
rocket trajectory bit-exact. The balloons come out of a Monte Carlo generation
that goes through scipy; the flight integration did not care.
One related observation, offered as information rather than as a complaint.
uv.lock carries two resolutions -- numpy 2.2.6 / scipy 1.15.3 for older
Python, 2.4.5 / 1.17.1 for newer -- while .python-version and the CI pin both
say only "3.14", and the CI matrix deliberately runs 3.10 and 3.14. Two
competitors who both installed from the lockfile can therefore be on different
numeric stacks. Our measurement says that does not reach the score, but the repo
does not currently say so anywhere, and a competitor who notices the split has no
way to tell whether it matters.
Your question
The qualification and final rounds have each team run on their own machine, with
seeds drawn on the day. That makes cross-machine numeric variation a property of
the format rather than a detail of anyone's setup, and I could not find anywhere
in the repo that says how much of it to expect.
We measured it because we needed the answer for our own workflow, and the number
turned out to be small enough that it may be worth stating in the docs. Two
questions:
rounds, or is a run on a competitor's machine taken as authoritative?
baseline section already explains BLAS jitter in the stored baselines, and
this is the same phenomenon reaching the score instead of the fixtures.
Context
Two machines, scenario 1, seed 0. The only environment difference we could find
was a Python patch release, 3.14.6 against 3.14.7. The full-precision digests of
the rocket trajectory differed; the balloon field matched.
A differing digest is not by itself a finding, since a difference of 1e-13 m and
a difference of 100 m produce the same one. So we measured against a
perturbation larger than a patch release instead, scipy 1.16.2 against 1.17.1:
Seven orders of magnitude below anything that moves a 1.5 m pop verdict. On that
evidence we treat runs from different machines as comparable and rank by a
multi-seed median rather than by digest equality.
The sensitivity ran opposite to our guess, which is the part we would not have
predicted from the code: a scipy bump moves the balloon field and leaves the
rocket trajectory bit-exact. The balloons come out of a Monte Carlo generation
that goes through scipy; the flight integration did not care.
One related observation, offered as information rather than as a complaint.
uv.lockcarries two resolutions -- numpy 2.2.6 / scipy 1.15.3 for olderPython, 2.4.5 / 1.17.1 for newer -- while
.python-versionand the CI pin bothsay only "3.14", and the CI matrix deliberately runs 3.10 and 3.14. Two
competitors who both installed from the lockfile can therefore be on different
numeric stacks. Our measurement says that does not reach the score, but the repo
does not currently say so anywhere, and a competitor who notices the split has no
way to tell whether it matters.