Skip to content

[NGT] Consolidation of HLT nano DQM - #51755

Open
mmusich wants to merge 12 commits into
cms-sw:masterfrom
mmusich:mm_run_dqm_for_hltnano
Open

[NGT] Consolidation of HLT nano DQM #51755
mmusich wants to merge 12 commits into
cms-sw:masterfrom
mmusich:mm_run_dqm_for_hltnano

Conversation

@mmusich

@mmusich mmusich commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

PR description:

This PR collects a miscellaneous sets of developments to have more thorough testing of the HLT nano flavour produced in the package HLTrigger/NGTScouting.
Main developments:

  • add more thorough coverage of the quantities in the hltNanoDQM
  • restore the unit test testProduceNanoHLT with fresh input data from CMSSW_20_0_0_pre1
  • add two new families of relval workflows with offsets .7592 and .7721 which test the case of HLT nano production out of FEVTDEBUGHLT input (from previous step) for both the HLT timing menu and the NGT scouting menu.
    • the workflows are added both in the ph2_hlt and nano matrices.

PR validation:

  • runTheMatrix.py -l ph2_hlt -i all --ibeos runs fine.
  • scram b runtests_testProduceNanoHLT runs fine.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

N/A

@cmsbuild cmsbuild added this to the CMSSW_20_1_X milestone Aug 25, 2026
@mmusich

mmusich commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

type ngt

@cmsbuild

cmsbuild commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

cms-bot internal usage

@mmusich

mmusich commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

enable nano

@mmusich

mmusich commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

test parameters:

  • workflows = ph2_hlt

@cmsbuild

Copy link
Copy Markdown
Contributor

@cmsbuild

Copy link
Copy Markdown
Contributor

A new Pull Request was created by @mmusich for master.

It involves the following packages:

  • Configuration/PyReleaseValidation (pdmv)
  • HLTrigger/NGTScouting (hlt)

@AdrianoDee, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @cmsbuild, @kfjack, @miquork, @mmusich, @sroychow can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @makortel, @missirol, @slomeo this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@mmusich

mmusich commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

@cmsbuild, please test

@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Size: This PR adds an extra 80KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0da494/55536/summary.html
COMMIT: d04af21
CMSSW: CMSSW_20_1_X_2026-08-25-1100/el9_amd64_gcc14
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51755/55536/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 10 lines to the logs
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 59
  • DQMHistoTests: Total histograms compared: 4089582
  • DQMHistoTests: Total failures: 8
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4089556
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 58 files compared)
  • Checked 249 log files, 203 edm output root files, 59 DQM output files
  • TriggerResults: no differences found

Nano size comparison Summary:

  • Nano ERROR: Missing ref/3000.0003_TTbar_14TeV+Run4D127_HLTPhase2WithNanoAndDQM-size.json
  • Nano ERROR: Missing ref/3000.0003_TTbar_14TeV+Run4D127_HLTPhase2WithNanoAndDQM-size.json
  • Nano ERROR: Missing ref/3000.0004_TTbar_14TeV+Run4D127_NGTScoutingWithNano-size.json
  • Nano ERROR: Missing ref/3000.0004_TTbar_14TeV+Run4D127_NGTScoutingWithNano-size.json
  • Nano ERROR: Missing ref/3000.0005_TTbar_14TeV+Run4D127_NGTScoutingWithNanoAndDQM-size.json
  • Nano ERROR: Missing ref/3000.0005_TTbar_14TeV+Run4D127_NGTScoutingWithNanoAndDQM-size.json
  • Nano ERROR: Missing ref/3000.0006_TTbar_14TeV+Run4D127_NGTScoutingWithNanoValid-size.json
  • Nano ERROR: Missing ref/3000.0006_TTbar_14TeV+Run4D127_NGTScoutingWithNanoValid-size.json
  • Nano ERROR: Missing ref/3000.0007_TTbar_14TeV+Run4D127_L1NGTScoutingWithNanoValid-size.json
  • Nano ERROR: Missing ref/3000.0007_TTbar_14TeV+Run4D127_L1NGTScoutingWithNanoValid-size.json
workflow id kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
ScoutingNANOdata124Xrun3 2500.1501 0.772 0.772 0.000 ( +0.0% ) 25.60 33.84 -24.3% 1.587 1.590
ScoutingNANOmonitordata124Xrun3 2500.1502 0.862 0.862 0.000 ( +0.0% ) 10.65 13.76 -22.6% 1.440 1.433
ScoutingNANOdata130Xrun3 2500.1503 1.251 1.251 0.000 ( +0.0% ) 19.31 25.08 -23.0% 1.566 1.562
ScoutingNANOmonitordata130Xrun3 2500.1504 1.297 1.297 0.000 ( +0.0% ) 8.73 10.57 -17.5% 1.439 1.436
ScoutingNANOdata140Xrun3 2500.2307 1.101 1.101 0.000 ( +0.0% ) 22.51 29.78 -24.4% 1.594 1.594
L1ScoutingSelectionNANOdata140Xrun3 2500.2311 0.081 0.081 0.000 ( +0.0% ) 33.27 49.63 -33.0% 0.861 0.859
L1ScoutingNANOdata140Xrun3 2500.231 0.082 0.082 0.000 ( +0.0% ) 3.05 4.19 -27.2% 0.961 0.956
muDPGNANOBkg140Xrun3 2500.2403 2.840 2.840 0.000 ( +0.0% ) 16.61 20.00 -16.9% 1.175 1.172
hcalDPGNANO140Xrun3 2500.2404 486.015 486.015 0.000 ( +0.0% ) 0.81 1.14 -28.6% 1.719 1.722
hcalDPGCalibNANO140Xrun3 2500.2405 826.418 826.418 0.000 ( +0.0% ) 1.05 1.50 -30.4% 1.709 1.696
ScoutingNANOdata150Xrun3 2500.3307 1.738 1.738 0.000 ( +0.0% ) 24.05 31.74 -24.2% 1.585 1.591
L1ScoutingNANOdata150Xrun3 2500.3312 0.078 0.078 0.000 ( +0.0% ) 2.63 3.67 -28.2% 0.955 0.954
L1ScoutingSelectionNANOdata150Xrun3 2500.3313 0.078 0.078 0.000 ( +0.0% ) 19.48 27.19 -28.4% 0.965 0.962
L1ScoutingNANOdata160Xrun3 2500.4301 0.110 0.110 0.000 ( +0.0% ) 1.64 2.19 -25.0% 0.943 0.942
L1ScoutingSelectionNANOdata160Xrun3 2500.4302 0.113 0.113 0.000 ( +0.0% ) 14.51 19.24 -24.6% 0.977 0.975
DYToLL_M-50_13TeV_pythia8 2500.9101 14.970 14.970 0.000 ( +0.0% ) 4.09 6.50 -37.1% 1.152 1.145
DYToll01234Jets_5f_LO_MLM_Madgraph_LHE_13TeV 2500.9102 0.240 0.199 0.041 ( +20.3% ) 1.20 3.59 -66.7% 0.853 0.869
TTbar_Pow_LHE_13TeV 2500.9103 0.110 0.110 0.000 ( +0.0% ) 1.85 2.73 -32.2% 0.858 0.877
TTbar_14TeV+Run4D127_HLTPhase2WithNano 3000.0001 10.884 10.888 -0.004 ( -0.0% ) 0.07 0.12 -37.3% 4.240 4.299
TTbar_14TeV+Run4D127_HLTPhase2WithNanoValid 3000.0002 203.475 203.479 -0.004 ( -0.0% ) 0.07 0.11 -33.0% 4.686 4.755

AMD_MI300X Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 82 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 8995
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 164744
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 62 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 7994
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 165745
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

NANO Comparison Summary

Summary:

  • You potentially added 2003 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 1
  • DQMHistoTests: Total histograms compared: 0
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 0
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
  • Checked 40 log files, 25 edm output root files, 1 DQM output files
  • TriggerResults: WARNING - found zero valid hltDiff JSON outputs to summarise (no outputs produced)

NVIDIA_H100 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 120 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 10769
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 162970
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

NVIDIA_L4 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 80 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 9185
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 164554
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 97 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 8580
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 165159
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 65 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 7072
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 166667
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff 37.2 exceeds +/- 30.0 MiB

@mmusich

mmusich commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

+hlt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants