Move Run 4 default to D127

[kpedro88]

PR description:

Followup to #51601: move the default for central MC production.

(NB: the slew of test scripts that are not part of unit tests are not updated here.)

PR validation:

runTheMatrix.py workflows are updated accordingly. 37634.0 was tested locally and runs successfully.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Will be backported to 20_0_X. (Requires #51620 to be merged and in an IB first.)

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51633/50465

• There are other open Pull requests which might conflict with changes you have proposed:
  • File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Added NPS DQM module #51535
  • File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): [Do not merge] For cms-bot PR testing #48637

[cmsbuild]

A new Pull Request was created by @kpedro88 for master.

It involves the following packages:

• CondTools/Geometry (db)
• Configuration/Geometry (geometry)
• Configuration/PyReleaseValidation (pdmv)
• Configuration/StandardSequences (operations)

@AdrianoDee, @Alejandro1400, @DickyChant, @Dr15Jones, @EltonSh, @JanChyczynski, @antoniovagnerini, @arunhep, @bsunanda, @civanch, @cmsbuild, @davidlange6, @fabiocos, @francescobrivio, @ftenchini, @kfjack, @kpedro88, @makortel, @mandrenguyen, @mdhildreth, @miquork, @perrotta, @sroychow can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @JanChyczynski, @Martin-Grunewald, @PonIlya, @VinInn, @VourMa, @bsunanda, @dgulhan, @elusian, @fabiocos, @felicepantaleo, @makortel, @mmasciov, @mmusich, @mtosi, @rovere, @sameasy, @slomeo, @vargasa this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[kpedro88]

please test

[cmsbuild]

-1

Failed Tests: RelVals
Size: This PR adds an extra 104KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-30e608/55108/summary.html
COMMIT: 4c832bd
CMSSW: CMSSW_20_1_X_2026-08-05-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51633/55108/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

• 37834.0A fatal system signal has occurred: segmentation violation
• 37634.911

----- Begin Fatal Exception 05-Aug-2026 20:43:31 CEST-----------------------
An exception of category 'DDFilteredView' occurred while
   [0] Processing  stream begin Run run: 1 stream: 0
   [1] Prefetching for module GEMPadDigiProducer/'simMuonGEMPadDigis'
   [2] Calling method for EventSetup module GEMGeometryESModule/'gemGeometry'
Exception Message:
Unknown boolean solid component
----- End Fatal Exception -------------------------------------------------

[kpedro88]

not sure what to do here, since we have not generated any RelVals with D127 yet. @cms-sw/pdmv-l2 @cms-sw/orp-l2

[mandrenguyen]

I'm not sure pdmv is available at the moment. If the errors are just caused by missing inputs, then we can just ignore them.

[AdrianoDee]

This is fine

[kpedro88]

I think the crash in 37834.0 is because of missing pileup input (though the stack trace points at L1NNTauProducer::~L1NNTauProducer(), which is strange).

The crash in the DD4hep workflow is more concerning. @bsunanda can you investigate?

[mmusich]

test parameters:

• enable = hlt_p2_integration
• workflows = ph2_hlt

[mmusich]

#51633 (comment) to avoid surprises from the HLT side

[kpedro88]

The full trace for the segmentation violation in 37834.0 is:

#0  0x00007fff201880b0 in ?? ()
#1  0x00007fff9b041443 in std::_Sp_counted_deleter<hls4mlEmulator::Model*, hls4mlEmulator::ModelLoader::load_model()::{lambda(hls4mlEmulator::Model*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ()
   from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-08-05-1100/external/el9_amd64_gcc13/lib/libemulator_interface.so
#2  0x00007fff2083d2ea in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fff27e88d20)
    at /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02953/el9_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/include/c++/13.4.0/bits/shared_ptr_base.h:346
#3  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fff27e88d20)
    at /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02953/el9_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/include/c++/13.4.0/bits/shared_ptr_base.h:317
#4  0x00007fff20883de1 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>, this=<optimized out>)
    at /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02953/el9_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/include/c++/13.4.0/bits/shared_ptr_base.h:1071
#5  std::__shared_ptr<hls4mlEmulator::Model, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>, this=<optimized out>)
    at /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02953/el9_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/include/c++/13.4.0/bits/shared_ptr_base.h:1524
#6  std::shared_ptr<hls4mlEmulator::Model>::~shared_ptr (this=<optimized out>, this=<optimized out>)
    at /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02953/el9_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/include/c++/13.4.0/bits/shared_ptr.h:175
#7  TauNNIdHW::~TauNNIdHW (this=<optimized out>, this=<optimized out>) at src/L1Trigger/Phase2L1ParticleFlow/interface/taus/TauNNIdHW.h:148
#8  std::default_delete<TauNNIdHW>::operator() (__ptr=0x7fff28b824b0, this=<optimized out>)
    at /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02953/el9_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/include/c++/13.4.0/bits/unique_ptr.h:99
#9  std::unique_ptr<TauNNIdHW, std::default_delete<TauNNIdHW> >::~unique_ptr (this=<optimized out>, this=<optimized out>)
    at /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02953/el9_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/include/c++/13.4.0/bits/unique_ptr.h:404
#10 L1NNTauProducer::~L1NNTauProducer (this=<optimized out>, this=<optimized out>) at src/L1Trigger/Phase2L1ParticleFlow/plugins/L1NNTauProducer.cc:352
#11 0x00007fff20883f1f in L1NNTauProducer::~L1NNTauProducer (this=<optimized out>, this=<optimized out>)
    at src/L1Trigger/Phase2L1ParticleFlow/plugins/L1NNTauProducer.cc:352

This looks like an instance of cms-hls4ml/hls4mlEmulatorExtras#7. I do not think it is directly related to this PR, but it is still concerning to encounter right before an MC campaign starts.

[makortel]

The exception comes via

#16 0x0000151bae717426 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::SignallingProductRegistryFiller&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) [clone .cold] () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02953/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-08-04-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so
#17 0x0000151bae81c358 in edm::ScheduleItems::initModules(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ModuleTypeResolverMaker const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02953/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-08-04-2300/lib/el9_amd64_gcc13/libFWCoreFramework.so

that smells like an exception being thrown in the Schedule constructor (that includes the construction of modules)

[kpedro88]

It might be related to early module destruction?

[makortel]

It might be related to early module destruction?

No (even if the stack trace shows deleteModulesEarly() function). The deletion of non-consumed modules is done later around the begin job phase.

I'd suggest to run the job in a debugger and catch exceptions (or run it through cmsTraceExceptions).

[mandrenguyen]

@kpedro88 is there a backport already? Maybe we can get that going in parallel?

[kpedro88]

See #51658

[perrotta]

+db

• In CondTools/Geometry the tests are updated to D127 coherently with the other updates in this PR

[AdrianoDee]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (test failures were overridden). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

[AdrianoDee]

@mandrenguyen yes, this is not strictly needed.

Also: when is the new release (for 20_1_X) expected? Just because with this in we will have to live with the input file error until we have it available to produce the new RelVals (+the time to produce them).

[mandrenguyen]

@mandrenguyen yes, this is not strictly needed.

Also: when is the new release (for 20_1_X) expected? Just because with this in we will have to live with the input file error until we have it available to produce the new RelVals (+the time to produce them).

We are overdue for a new 20_1_X pre-release (pre2). Should we go ahead and build the pre-release and then merge this after? Or should we include it in pre2?

[mandrenguyen]

+1
After discussion with @cms-sw/pdmv-l2 better update the default and build pre2 ASAP

[mmusich]

Also: when is the new release (for 20_1_X) expected? Just because with this in we will have to live with the input file error until we have it available to produce the new RelVals (+the time to produce them).

can we remove the failing workflow from the PR matrix tests until the sample exists?

[makortel]

Also: when is the new release (for 20_1_X) expected? Just because with this in we will have to live with the input file error until we have it available to produce the new RelVals (+the time to produce them).

can we remove the failing workflow from the PR matrix tests until the sample exists?

I'd argue the failing workflow must be removed because the current failure prevents DQM and Reco comparisons from running and therefore prevents assessing whether PRs physics result impact is along expectations.

[mmusich]

I'd argue the failing workflow must be removed because the current failure prevents DQM and Reco comparisons from running and therefore prevents assessing whether PRs physics result impact is along expectations.

This is indeed what I meant, as it seems no actions were taken so far, I went ahead at #51667

[makortel]

For the next time, could I suggest something along the following for the update process

• set of workflows for the upcoming default geometry are added to be run in IBs and PR tests
  • workflows for current default geometry are kept
• prerelease is built
• RelVals are produced for the upcoming default geometry
• default geometry is updated with those RelVal files as (pileup) input
  • workflows for the previous default geometry can be removed if no longer needed

[kpedro88]

the current failure prevents DQM and Reco comparisons

This seems like an avoidable failure of our CI. Comparisons can be skipped for failing workflows and still generated for passing workflows.

For the next time, could I suggest something along the following for the update process

Primarily, there needs to be a defined responsibility, assigned to a specific coordination / L2 area, to organize the timeline and process. Just having it written down is not sufficient. This responsibility used to be part of the L2 Upgrade Software Coordinator position and was not reassigned when that position was eliminated.

More broadly, CMSSW testing requires both shorter and longer timelines than RelVals: we need samples quickly, and then we need them to last for a long time. In the long term, we should disconnect these two things because of the unavoidable impedance mismatch (or otherwise significantly reform the operational aspects).

[AdrianoDee]

Primarily, there needs to be a defined responsibility, assigned to a specific coordination / L2 area

At the cost of shooting myself in the foot, I think this is "PdmV" / OV responsibility now. And the procedure proposed by Matti is for sure the right one. This was what I was mentioning above.

Then, given the urgency driven by simulation and geometry, and the fact that the new release was coming shortly after, I think it was worth going ahead and merging this to solve the problem in ~24h, instead of doing a series of small PRs one after the other. After all we are in mid-August, and the PR rate is pretty low compared to usual; personally I supposed we could live with 24h of one failing wf.

The ultimate solution is indeed already available in #51684.

Then I agree with @kpedro88 about the fact that:

1. if we can't see the DQM and Reco comparisons with one wf failing, this is a flaw of the current CI;
2. all this may be decoupled from the RelVal production itself. Mostly because in this case we had a release basically ready to be cut. But this does not happen all the time.