upgrade ACTS from v36 to v47

[bloodyyugo]

Upgrade ACTS versions for 36-->47. Closes #2100

---

Algorithm changes that may effect tracks:

• CKF branch handling
  The CKF received improvements to outlier rejection and branch pruning
  across these 11 major versions. This can shift the
  efficiency/fake-rate tradeoff even with identical inputs.

• GainMatrixSmoother → smoothTrack()
  The Kalman backward (RTS) smoother was rewritten as a free function.
  The math is identical in principle, but the implementation details
  (handling of degenerate states, track-end boundary conditions, numerical
  precision) may differ. Most likely source of small systematic shifts in
  momentum resolution or pull distributions.

• EigenStepper / RKN propagator
  Step-size control and Jacobian transport have had stability improvements.
  Effects appear mainly in track-to-truth residuals for long extrapolations
  (e.g., tagger perigee → ECAL scoring plane).

• GSF Bethe-Heitler handling
  AtlasBetheHeitlerApprox was renamed PolynomialBetheHeitlerApprox — the
  polynomial coefficients are the same. However, the GSF received active
  development between v36 and v47 in component reduction strategies, weight
  re-normalization, and handling of near-degenerate Gaussian mixtures. For
  LDMX, where the GSF drives electron momentum measurement, this is the
  highest-risk area for physics-level differences vs. trunk.

---

Interface changes etc:

• In v46, the calibrator and measurementSelector
  delegates moved out of CombinatorialKalmanFilterExtensions into a new
  Acts::TrackStateCreator object. The SourceLinkAccessorDelegate also moved there.

• DetectorElementBase -> Acts::SurfacePlacementBase.

• transform() override -> localToGlobalTransform() (matches new base class).

• surface->assignDetectorElement() -> surface->assignSurfacePlacement().

• thickness() is no longer a pure virtual in the base; override removed.

• isSensitive() added as a required override (returns true for sensor elements).

• Acts::ViewConfig now uses C++ designated initializers: {.color = {r, g, b}}.

• Added tagger_start_surface_ at ACTS x=-617mm (1mm inside the tagger
  volume outer boundary at x=-618mm, 1.5mm upstream of the L1 doublet at
  x=-615.5mm).

• Recoil tracking volume extended upstream so its low-x edge is at x=-1mm,
  placing the target plane (x=0) clearly inside the volume for the CKF Navigator
  (previously the target was on or outside the volume boundary, causing Navigator
  failures for recoil tracks).

• BetheHeitlerApprox changed from a local stack object to a shared_ptr:
  Old: BetheHeitlerApprox bh = Acts::makeDefaultBetheHeitlerApprox();
  New: auto bh = std::make_sharedActs::AtlasBetheHeitlerApprox(
  Acts::makeDefaultBetheHeitlerApprox());

• GsfOptions.propagatorPlainOptions now requires an explicit cast:
  gsf_options.propagatorPlainOptions =
  static_castActs::PropagatorPlainOptions(propagator_options);

• A separate GsfExtrapPropagator (EigenStepper + VoidNavigator) was added for
  post-fit track extrapolations (to target, ECAL). VoidNavigator is required
  because the ECAL scoring plane is outside the tracking geometry volume and the
  standard Navigator would stop propagation at the volume boundary.

• GainMatrixSmoother removed; Acts::smoothTrack() used instead, with its return
  value checked.

• Acts::FreeSquareMatrix renamed (TrackingUtils.cxx)
  Old: Acts::FreeSquareMatrix
  New: Acts::FreeMatrix
  (defined in Acts/Definitions/TrackParametrization.hpp, same as before)

• Acts::GeometryContext default constructor made private
  The no-arg Acts::GeometryContext() constructor is private in v47. Code that
  held Acts::GeometryContext as a member and relied on implicit default
  initialization would fail to compile.

[tomeichlersmith]

For LDMX, where the GSF drives electron momentum measurement, this is the
highest-risk area for physics-level differences vs. trunk.

Do you plan to do an initial study of these differences and post them here before the PR is merged? Or do you think it is too detailed (and the differences are likely improvements) so it will wait for future work?

[github-actions]

Validation Results

Some validation samples failed! ❌

Sample	Status
cascade_history	✅ PASS
deep_ecal_gun	✅ PASS
hcal	✅ PASS
eat_signal	❌ FAIL (46 histograms failed KS test) (artifact)
ecal_pn	❌ FAIL (135 histograms failed KS test) (artifact)
inclusive	❌ FAIL (200 histograms failed KS test) (artifact)
it_pileup	❌ FAIL (271 histograms failed KS test) (artifact)
kaon_enhanced	❌ FAIL (115 histograms failed KS test) (artifact)
reduced_ldmx	❌ FAIL
signal	❌ FAIL (221 histograms failed KS test) (artifact)
signal_target_al	❌ FAIL (452 histograms failed KS test) (artifact)
target_genie	❌ FAIL (110 histograms failed KS test) (artifact)
target_pn_lyso	❌ FAIL (323 histograms failed KS test) (artifact)
target_ti_en	❌ FAIL (138 histograms failed KS test) (artifact)
wab_lhe	❌ FAIL (101 histograms failed KS test) (artifact)

eat_signal:

• 46 plots failed the KS test against gold.
• Text Differences Between Logs (9923 lines differ)
• Log character count differs by 61%: gold=1140607, new=1844041
• Timing for eat_signal: gold=1618s, new=1576s
• Timing within 2% of gold (tolerance 10%)

ecal_pn:

• 135 plots failed the KS test against gold.
• Text Differences Between Logs (55772 lines differ)
• Log character count differs by 1%: gold=24115214, new=24548294
• Timing for ecal_pn: gold=6718s, new=6702s
• Timing within 0% of gold (tolerance 10%)

inclusive:

• 200 plots failed the KS test against gold.
• Text Differences Between Logs (61584 lines differ)
• Log character count differs by 2%: gold=21025892, new=21578617
• Timing regression for inclusive: new=6643s vs gold=3573s (+85%, tolerance 10%)
• Timing for inclusive: gold=3573s, new=6643s

it_pileup:

• 271 plots failed the KS test against gold.
• Text Differences Between Logs (24971 lines differ)
• Log character count differs by 9%: gold=9752960, new=8841824
• Timing for it_pileup: gold=599s, new=639s
• Timing within 6% of gold (tolerance 10%)

kaon_enhanced:

• 115 plots failed the KS test against gold.
• Text Differences Between Logs (26747 lines differ)
• Log character count differs by 1%: gold=10113224, new=10302254
• Timing for kaon_enhanced: gold=3095s, new=3012s
• Timing within 2% of gold (tolerance 10%)

signal:

• 221 plots failed the KS test against gold.
• Text Differences Between Logs (55974 lines differ)
• Log character count differs by 2%: gold=16413683, new=16778912
• Timing for signal: gold=1189s, new=1205s
• Timing within 1% of gold (tolerance 10%)

signal_target_al:

• 452 plots failed the KS test against gold.
• Text Differences Between Logs (46203 lines differ)
• Log character count differs by 2%: gold=16496870, new=16986051
• Timing for signal_target_al: gold=936s, new=942s
• Timing within 0% of gold (tolerance 10%)

target_genie:

• 110 plots failed the KS test against gold.
• Text Differences Between Logs (1895 lines differ)
• Log character count differs by 7%: gold=1406660, new=1513713
• Timing regression for target_genie: new=9736s vs gold=5707s (+70%, tolerance 10%)
• Timing for target_genie: gold=5707s, new=9736s

target_pn_lyso:

• 323 plots failed the KS test against gold.
• Text Differences Between Logs (40675 lines differ)
• Log character count differs by 2%: gold=20168222, new=20743639
• Timing for target_pn_lyso: gold=6341s, new=6324s
• Timing within 0% of gold (tolerance 10%)

target_ti_en:

• 138 plots failed the KS test against gold.
• Text Differences Between Logs (877 lines differ)
• Log character count differs by 1%: gold=393174, new=398437
• Timing for target_ti_en: gold=2738s, new=2776s
• Timing within 1% of gold (tolerance 10%)

wab_lhe:

• 101 plots failed the KS test against gold.
• Text Differences Between Logs (19130 lines differ)
• Log character count differs by 4%: gold=13180440, new=12553398
• Timing for wab_lhe: gold=6754s, new=6830s
• Timing within 1% of gold (tolerance 10%)

cascade_history:

• Text Differences Between Logs (2 lines differ)
• Timing for cascade_history: gold=6217s, new=5981s
• Timing within 3% of gold (tolerance 10%)

deep_ecal_gun:

• Text Differences Between Logs (24 lines differ)
• Timing for deep_ecal_gun: gold=3763s, new=4129s
• Timing within 9% of gold (tolerance 10%)

hcal:

• Text Differences Between Logs (2 lines differ)
• Timing regression for hcal: new=853s vs gold=665s (+28%, tolerance 10%)
• Timing for hcal: gold=665s, new=853s

[bloodyyugo]

The most recent commit fixes the Ecal tracking. There were two issues, first the starting surface needed to be in the tracking volume (same thing we saw for recoil and tagger); also the surfaces given to ACTs didn't have a "sensitive" surface defined, so the tracking didn't know which surfaces the hits were supposed to be on (and what it should be swimming track candidates to).

So now we see tracks for ECal tracking...a few of the plots still fail the validation tests but looking at them they all look very close.

For the trackers, the only thing that sticks out to me is the ecal_pulls_of_loc0 (so, x position pull at the ecal). It's much broader for acts47 than the trunk. I've attached the plot RecoilGASDQM_ecal_Pulls_of_loc0.pdf.

Note, I think it's best to look at the "GAS" plots (i.e. after getting rid of highly redundant tracks). The raw ones, e.g. "RecoilDQM" I expect to be different because the changes to "outlier rejection and branch pruning" while there have been many changes to the GSF algorithm. For the recoil GAS tracks, only plots related to the loc0 pulls fail. For the tagger, the errors on the track parameters (not the parameter values themselves) change a bit so the _error and _pull plots fail the KS tests. It is a bit weird that this only shows up in the tagger and not the recoil though.

[bloodyyugo]

/run-validation

[github-actions]

The validation workflow is running here: https://github.com/LDMX-Software/ldmx-sw/actions/runs/30561741919.

[github-actions]

Validation Results

Some validation samples failed! ❌

Sample	Status
cascade_history	✅ PASS
deep_ecal_gun	✅ PASS
hcal	✅ PASS
eat_signal	❌ FAIL (9 histograms failed KS test) (artifact)
ecal_pn	❌ FAIL (135 histograms failed KS test) (artifact)
inclusive	❌ FAIL (200 histograms failed KS test) (artifact)
it_pileup	❌ FAIL (271 histograms failed KS test) (artifact)
kaon_enhanced	❌ FAIL (115 histograms failed KS test) (artifact)
reduced_ldmx	❌ FAIL
signal	❌ FAIL (188 histograms failed KS test) (artifact)
signal_target_al	❌ FAIL (452 histograms failed KS test) (artifact)
target_genie	❌ FAIL (110 histograms failed KS test) (artifact)
target_pn_lyso	❌ FAIL
target_ti_en	❌ FAIL
wab_lhe	❌ FAIL

eat_signal:

• 9 plots failed the KS test against gold.
• Timing for eat_signal: gold=1618s, new=1609s
• Timing within 0% of gold (tolerance 10%)

ecal_pn:

• 135 plots failed the KS test against gold.
• Text Differences Between Logs (57404 lines differ)
• Log character count differs by 8%: gold=24115214, new=26263133
• Timing for ecal_pn: gold=6718s, new=6795s
• Timing within 1% of gold (tolerance 10%)

inclusive:

• 200 plots failed the KS test against gold.
• Text Differences Between Logs (61620 lines differ)
• Log character count differs by 11%: gold=21025892, new=23452406
• Timing regression for inclusive: new=6745s vs gold=3573s (+88%, tolerance 10%)
• Timing for inclusive: gold=3573s, new=6745s

it_pileup:

• 271 plots failed the KS test against gold.
• Text Differences Between Logs (24969 lines differ)
• Log character count differs by 9%: gold=9752960, new=8841824
• Timing for it_pileup: gold=599s, new=612s
• Timing within 2% of gold (tolerance 10%)

kaon_enhanced:

• 115 plots failed the KS test against gold.
• Text Differences Between Logs (27944 lines differ)
• Log character count differs by 9%: gold=10113224, new=11121319
• Timing for kaon_enhanced: gold=3095s, new=2961s
• Timing within 4% of gold (tolerance 10%)

signal:

• 188 plots failed the KS test against gold.
• Text Differences Between Logs (56553 lines differ)
• Log character count differs by 12%: gold=16413683, new=18410869
• Timing for signal: gold=1189s, new=1195s
• Timing within 0% of gold (tolerance 10%)

signal_target_al:

• 452 plots failed the KS test against gold.
• Text Differences Between Logs (46197 lines differ)
• Log character count differs by 2%: gold=16496870, new=16986051
• Timing for signal_target_al: gold=936s, new=921s
• Timing within 1% of gold (tolerance 10%)

target_genie:

• 110 plots failed the KS test against gold.
• Text Differences Between Logs (1895 lines differ)
• Log character count differs by 7%: gold=1406660, new=1513713
• Timing regression for target_genie: new=7507s vs gold=5707s (+31%, tolerance 10%)
• Timing for target_genie: gold=5707s, new=7507s

cascade_history:

• Text Differences Between Logs (2 lines differ)
• Timing for cascade_history: gold=6217s, new=6222s
• Timing within 0% of gold (tolerance 10%)

deep_ecal_gun:

• Text Differences Between Logs (20 lines differ)
• Timing for deep_ecal_gun: gold=3763s, new=4155s
• Timing within 10% of gold (tolerance 10%)

hcal:

• Text Differences Between Logs (2 lines differ)
• Timing regression for hcal: new=844s vs gold=665s (+26%, tolerance 10%)
• Timing for hcal: gold=665s, new=844s

[tvami]

Thanks @bloodyyugo !

[tvami]

Hmm should have paid more attention to the CI
#2114