Speed up runTheMatrix.py by up to 7x - #51654
Open
felicepantaleo wants to merge 2 commits into
Open
Conversation
Ask a special upgrade workflow's condition() before building its step list, skip the flavors that customise no step, make merge() iterative, build a fragment step's dictionary on first read, and reuse a step's cmsDriver command across the workflows that share it. runTheMatrix.py -n goes from 26.0 s to 9.0 s and -n -w upgrade from 30.3 s to 11.3 s, peak memory for -n from 1.42 GB to 0.77 GB. Output is byte identical over 36 invocations covering every matrix file, --show, --raw, --dryRun, --interactive and --checkInputs.
…ly what -l selects The upgrade matrix defines 2.4 M fragment steps in some 196 families and a run reads a few percent of them. Every name a family defines ends in the step name, so a family is built the first time one of its names is looked up, and readMatrix expands only the workflows -l selects. runTheMatrix.py -n goes from 9.0 s to 4.5 s and -n -l <workflow> from 9.0 s to 4.1 s, peak memory for -n from 0.77 GB to 0.32 GB. With -l, "found a total of N workflows" now reports the selected workflows, and the duplicate name notices cover only those.
Contributor
|
cms-bot internal usage |
Contributor
Author
|
@waredjeb @AuroraPerego fyi this is something we were looking at together last year. |
Contributor
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51654/50518
|
Contributor
Author
|
type ngt |
Contributor
|
A new Pull Request was created by @felicepantaleo for master. It involves the following packages:
@AdrianoDee, @DickyChant, @antoniovagnerini, @cmsbuild, @kfjack, @miquork, @sroychow can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
Contributor
Author
|
type improvements |
Contributor
Author
|
ignore tests-rejected with ib-failure |
Contributor
Author
|
@cmsbuild please test |
Contributor
|
+1 Size: This PR adds an extra 100KB to repository Comparison SummarySummary:
|
Contributor
|
enable gpu,threading,profiling,nano,rntuple,hlt_p2_timing |
Contributor
|
please test |
Contributor
|
-1 Failed Tests: RelVals-AMD_W7900 HLT P2 Timing: chart Failed RelVals-AMD_W7900
Comparison SummarySummary:
Nano size comparison Summary:
NANO Comparison SummarySummary:
NVIDIA_H100 Comparison SummarySummary:
NVIDIA_L40S Comparison SummarySummary:
NVIDIA_T4 Comparison SummarySummary:
RNTUPLE Comparison SummarySummary:
Max Memory Comparisons exceeding threshold NVIDIA_H100@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold: Expand to see workflows ...
Max Memory Comparisons exceeding threshold RNTUPLE@cms-sw/core-l2 , I found 120 workflow step(s) with memory usage exceeding the error threshold: Expand to see workflows ...
|
Contributor
|
test parameters:
|
Contributor
|
@cmsbuild, please test |
Contributor
|
-1 Failed Tests: RelVals-AMD_MI300X HLT P2 Timing: chart Failed RelVals-AMD_MI300X
Comparison SummarySummary:
Nano size comparison Summary:
AMD_W7900 Comparison SummarySummary:
NANO Comparison SummarySummary:
NVIDIA_H100 Comparison SummarySummary:
NVIDIA_L40S Comparison SummarySummary:
NVIDIA_T4 Comparison SummarySummary:
RNTUPLE Comparison SummarySummary:
Max Memory Comparisons exceeding threshold NVIDIA_H100@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold: Expand to see workflows ...
Max Memory Comparisons exceeding threshold RNTUPLE@cms-sw/core-l2 , I found 120 workflow step(s) with memory usage exceeding the error threshold: Expand to see workflows ...
|
This was referenced Aug 11, 2026
Merged
Merged
This was referenced Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
runTheMatrix.pyspends almost all of its time building the upgrade matrix at import, and most of that work is thrown away. Two independent commits, the first with byte-identical output, the second changing only what-lprints.1. Do not build what is then discarded (no behaviour change)
relval_upgrade.pybuilt a full step list for each of the ~1.5 M (upgrade key, fragment, special workflow) combinations and only afterwards askedcondition()whether to keep it. Of the 69condition()implementations onlyUpgradeWorkflow_DigiNoHLTlooks at the step list, soUpgradeWorkflowgains a class attributeconditionUsesStepList(Falseby default,TrueonDigiNoHLT) and the caller now asks first and builds the list only for the flavors that are accepted. A flavor that customises no step ends up with the baseline list, whichworkflow_()drops as spurious anyway, so it is skipped.MatrixUtil.merge()becomes iterative instead of recursive (same values, same key order, same result type, 5.78 M calls down to 2.07 M), a fragment step's dictionary is assembled the first time it is read rather than up front,Steps.__setitem__andMatrix.__setitem__no longer allocate a throwaway dict per insertion, andMatrixReaderreuses a step's cmsDriver command across the workflows that run it (438708 step instances resolve to 37317 distinct step names, and no upgrade workflow carries overrides or an added command).2. Build the step families on demand and expand only what
-lselectsThe upgrade matrix defines 2,388,313 step entries in 196 families, one per step name across every fragment and every upgrade key, and a run reads a few percent of them. Every name a family defines ends in
_<step name>, so the family a name belongs to is known from the name alone and can be built the first time one of its names is looked up. After the change 210,362 entries are built eagerly and the rest only when asked for.MatrixReaderadditionally expands only the workflows given to-l, so a single selected workflow touches a single family.Measurements
Host release
CMSSW_20_1_X_2026-08-06-1200, single run each, same machine:-n-n -l 12834.0-n -w upgrade -l 29634.0-n -l limited-n -w upgrade--interactivePeak resident memory for
-ngoes from 1.42 GB to 0.32 GB, and for-n -l <workflow>from 1.42 GB to 0.28 GB.Behaviour changes
None in the first commit. In the second commit, and only when
-lis given:found a total of N workflowsnow reports the number of selected workflows, since the others are no longer expanded.--checkInputschecks only the selected workflows, and the duplicate name notices fromcreateWorkFlowscover only them.--rawdeliberately keeps the old unfiltered path.The insertion order of the
stepsdictionary changes, because a family is appended when it is first touched. Nothing prints or depends on that order: the only two functions that iterate it,changeRefReleaseandaddForAll, act per entry.Validation
Two independent checks against the unmodified release, run separately for each commit.
workflows(111,268 for the upgrade matrix alone, with names, overrides, step lists and insertion order) plus the entirestepsdictionary (2,388,313 entries with per-step key order and values). The first commit reproduces it byte for byte including thestepsinsertion order; the second reproduces it byte for byte with workflows in insertion order and steps sorted by name.--show,-e,-lwith single workflows and with predefined sets,--raw,--step1,-i,--maxSteps,--command,--use-rntuple,--noCafVeto,--dryRun,--checkInputsand--interactive, comparing 112 captured files. For the first commit 109 of 112 are byte-identical, the three exceptions being tracebacks that quote a source line number ofMatrixReader.py, which grew by three lines. For the second commit the only further differences are the eight-linvocations, and in each of them the sole changed line is thefound a total of N workflowscount.