Skip to content

Print more details when regression tests fails - #342

Merged
Luthaf merged 1 commit into
metatensor:metatomic-corefrom
Luthaf:fix-regtests
Sep 30, 2026
Merged

Luthaf merged 1 commit into
metatensor:metatomic-corefrom
Luthaf:fix-regtests

Conversation

@Luthaf

@Luthaf Luthaf commented Sep 30, 2026

Copy link
Copy Markdown
Member

This should help debug failures when running the same test on different CI runners

Contributor (creator of pull-request) checklist

  • Tests updated (for new features and bugfixes)?
  • Documentation updated (for new features)?
  • Issue referenced (for PRs that solve an issue)?

Reviewer checklist

  • CHANGELOG updated with public API or any other important changes?

@Luthaf

Luthaf commented Sep 30, 2026 •

Copy link
Copy Markdown
Member Author

On some runners (https://github.com/metatensor/metatomic/actions/runs/36716334056/job/109891461253), this showed failures like

===== mismatches for 'mtt::cs_iso' (rtol=1e-04, atol=1e-04) =====
key=(o3_lambda=1, o3_sigma=-1) 1/24 elements not close, max abs diff=0.0004
    sample=(system=0, atom=7) components=[(o3_mu=-1)] property=(shielding=0): actual=2.85514 expected=2.85554 diff=0.0004

And

===== mismatches for 'energy' (rtol=1e-06, atol=1e-05) =====
key=(_=0) gradient 'strain' 2/9 elements not close, max abs diff=1.3e-05
    sample=(sample=0) components=[(xyz_1=0), (xyz_2=2)] property=(energy=0): actual=-1.79819 expected=-1.79821 diff=1.3e-05
    sample=(sample=0) components=[(xyz_1=2), (xyz_2=0)] property=(energy=0): actual=-1.78524 expected=-1.78525 diff=1.3e-05

In all cases, this seems to be coming from the fact that we run the models in float32 with fairly tight tolerances, so the fix is to loosen the tolerances a bit.

@Luthaf
Luthaf requested a review from EricBoittier September 30, 2026 12:54
"reference": "energy.mts",
"rtol": 1e-6,
"atol": 1e-5
"rtol": 1e-5,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@EricBoittier EricBoittier left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice this will be very valuable going forward.

@Luthaf
Luthaf merged commit 5ec31c2 into metatensor:metatomic-core Sep 30, 2026
22 checks passed
@Luthaf
Luthaf deleted the fix-regtests branch September 30, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants