During build, correctly set LD_LIBRARY_PATH for edm check of GPU libs

[smuzaffar]

During building, scram runs various edm checks e.g. edm write config, edm class version check or edm transient class for generated libraries. Edm checks fail when a library needs specific libs e.g. pytorch has 3 builds ( one for cpu only, only for cuda+cpu and one for rocm+cpu) and to load a alpaka/rocm shared lib scram should make sure that pytorch/rocm libs comes first. This PR dynamically change the LD_LIBRARY_PATH for each edm check and remove the unneeded lib paths.

We still have one issue that we run edmPluginRefresh at the end and this loads all cmssw libs in single process. I think we need to change this such that we run edmPluginRefresh multiple times providing a filter to it so that it only loads selected libs. SCRAM can then do

LD_LIBRARY_PATH=<rocm-libs>  edmPluginRefresh -p /path -f '*ROCmAsync'
LD_LIBRARY_PATH=<cuda-libs/cpu-libs>  edmPluginRefresh -p /path -e '*ROCmAsync'
``

`-f filter` is to only load libraries which matche `filter`
`-e filter1,filter2` is to load every thing except those which match `filter1` and `filter2`

[cmsbuild]

A new Pull Request was created by @smuzaffar for branch scramv3.

@akritkbehera, @cmsbuild, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[cmsbuild]

Pull request #121 was updated.

[cmsbuild]

Pull request #121 was updated.

[smuzaffar]

before merging it to default, I have created a tag V09-10-00 which we can use to test in DEVEL_X IBs

[cmsbuild]

Pull request #121 was updated.

[smuzaffar]

Enable gpu

[smuzaffar]

Please test using full cmssw

[cmsbuild]

Pull request #121 was updated.

[smuzaffar]

enable gpu,multi_microarchs

[smuzaffar]

Please test using full cmssw

[smuzaffar]

Please test using full cmssw

[cmsbuild]

Pull request #121 was updated.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a51a2/55772/summary.html
COMMIT: 2253adc
CMSSW: CMSSW_20_1_X_2026-09-08-2300/el9_amd64_gcc14
Additional Tests: GPU,MULTI_MICROARCHS,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw-config/121/55772/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @cms-sw [UBSAN] Apply abseil ubsan fix cmsdist#10837
• @CMS-HGCAL HGCAL: Fixes to dense index for SiPM-on-tile + additional FED unpacking flags cmssw#51815
• @JanGerritSchulz Remove trimmed tracking configuration for HLT Phase-2 cmssw#51816
• @ghyls Fix some tests in HeterogeneousCore/MPICore cmssw#51820
• @Electricks94 Remove SCALARS from SoA elements cmssw#51821
• @mmusich light modernization of HLTriggerOffline/BTag cmssw#51833
• @Annnnya Call MPI_Abort on error cmssw#51839
• @bsunanda Phase2-368U Add a new method in HGCSiliconDetId to modify u,v coordintes of wafer cmssw#51840

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a51a2/55772/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a51a2/55772/git-merge-result

Comparison Summary

Summary:

• You potentially removed 165 lines from the logs
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 45
• DQMHistoTests: Total histograms compared: 3592269
• DQMHistoTests: Total failures: 38
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3592213
• DQMHistoTests: Total skipped: 18
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
• Checked 195 log files, 163 edm output root files, 45 DQM output files
• TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

• You potentially removed 25 lines from the logs
• Reco comparison results: 96 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 173739
• DQMHistoTests: Total failures: 8761
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 164978
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

• You potentially removed 25 lines from the logs
• Reco comparison results: 74 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 173739
• DQMHistoTests: Total failures: 9031
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 164708
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

• You potentially removed 25 lines from the logs
• Reco comparison results: 78 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 173739
• DQMHistoTests: Total failures: 8982
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 164757
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

• You potentially removed 25 lines from the logs
• Reco comparison results: 63 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 173739
• DQMHistoTests: Total failures: 8610
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 165129
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

• You potentially removed 25 lines from the logs
• Reco comparison results: 62 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 173739
• DQMHistoTests: Total failures: 8372
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 165367
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: no differences found

Max Memory Comparisons exceeding threshold NVIDIA_H100

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

• Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff 191.8 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

• Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff -190.2 exceeds +/- 30.0 MiB