Skip to content

Introspection for SoABlocks - #51670

Merged
cmsbuild merged 2 commits into
cms-sw:masterfrom
Electricks94:SoABlocksIntrospection
Aug 27, 2026
Merged

Introspection for SoABlocks#51670
cmsbuild merged 2 commits into
cms-sw:masterfrom
Electricks94:SoABlocksIntrospection

Conversation

@Electricks94

Copy link
Copy Markdown
Contributor

PR description:

This PR extends the SoABlocks Layout for the introspection functionality. Normal SoALayouts can be printed using the stream insertion operator. In this way the total size in bytes as well as the byte size for each column and the used padding can be viewed. With this PR the stream insertion operator can also be used for SoABlocks which will output the size of each layout that is contained in the blocks and then the total size of the SoABlocks Layout.

PR validation:

This PR is validated by extending the SoAStreamInternal_t test case. In this test case an expected output stream for a specific layout is hardcoded and the output of the stream insertion operator is then checked

FYI @felicepantaleo

@cmsbuild

cmsbuild commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

cms-bot internal usage

@cmsbuild

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51670/50542

@cmsbuild

Copy link
Copy Markdown
Contributor

A new Pull Request was created by @Electricks94 for master.

It involves the following packages:

  • DataFormats/SoATemplate (heterogeneous)

@cmsbuild, @fwyzard, @makortel can you please review it and eventually sign? Thanks.
@missirol, @mmusich, @rovere this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@fwyzard

fwyzard commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

enable gpu

@fwyzard

fwyzard commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

please test

@fwyzard

fwyzard commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

+heterogeneous

@cmsbuild

Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs after it passes the integration tests. This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-83c72b/55253/summary.html
COMMIT: cbf5dbb
CMSSW: CMSSW_20_1_X_2026-08-11-1100/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51670/55253/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

@fwyzard

fwyzard commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

The GPU workflows failed with

ValueError: Undefined workflows: 34434.403, 34434.713, 34434.404, 34434.712, 34434.402, 34434.7503, 34434.751
+ return 1
+ exit 1
ERROR Running runTheMatrix for '-l 34434.402,34434.403,34434.404,34434.712,34434.713,34434.7503,34434.751'

Is something wrong with the matrix?

@mmusich

mmusich commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Is something wrong with the matrix?

My guess is that after #51633 this

https://github.com/cms-sw/cms-bot/blob/1c0a09a89aceb7892d8ec136e67dfd88d4f27ccd/cmssw-pr-test-config#L31

Needs an update

@fwyzard

fwyzard commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Ah.

So maybe

(NB: the slew of test scripts that are not part of unit tests are not updated here.)

was not such a good idea?

@fwyzard

fwyzard commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

@cms-sw/all-l2 can we please stop breaking the PR tests with updates to the geometry ?

@fwyzard

fwyzard commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

+heterogeneous

@cmsbuild

Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs after it passes the integration tests. This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @ftenchini, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-sw/cms-bot#2823

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Failed Tests: RelVals-AMD_MI300X amd_mi300xUnitTests
Size: This PR adds an extra 28KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-83c72b/55269/summary.html
COMMIT: 06b7db5
CMSSW: CMSSW_20_1_X_2026-08-11-2300/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51670/55269/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-83c72b/55269/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-83c72b/55269/git-merge-result

Failed RelVals-AMD_MI300X

  • 37634.40437634.404_TTbar_14TeV+Run4D127_Patatrack_PixelOnlyAlpaka_Profiling/step2_TTbar_14TeV+Run4D127_Patatrack_PixelOnlyAlpaka_Profiling.log
  • 37634.75137634.751_TTbar_14TeV+Run4D127_HLT75e33TimingAlpaka/step2_TTbar_14TeV+Run4D127_HLT75e33TimingAlpaka.log
  • 37634.750337634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid/step2_TTbar_14TeV+Run4D127_HLTHeterogeneousValid.log

Comparison Summary

Summary:

  • You potentially removed 1 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 44
  • DQMHistoTests: Total histograms compared: 3433258
  • DQMHistoTests: Total failures: 39
  • DQMHistoTests: Total nulls: 2
  • DQMHistoTests: Total successes: 3433199
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 9591.67 KiB( 43 files compared)
  • DQMHistoSizes: changed ( 18434.0,... ): 1598.605 KiB HLT/SecondaryVertices
  • DQMHistoSizes: changed ( 18434.0,... ): 0.004 KiB MessageLogger/Errors
  • DQMHistoSizes: changed ( 18434.0,... ): 0.004 KiB MessageLogger/Warnings
  • Checked 191 log files, 159 edm output root files, 44 DQM output files
  • TriggerResults: found differences in 6 / 43 workflows

NVIDIA_H100 Comparison Summary

Summary:

  • You potentially removed 2 lines from the logs
  • Reco comparison results: 111 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 172823
  • DQMHistoTests: Total failures: 11962
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 160861
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 3197.21 KiB( 6 files compared)
  • DQMHistoSizes: changed ( 37634.7503,... ): 1598.605 KiB HLT/SecondaryVertices
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: found differences in 2 / 6 workflows

NVIDIA_L40S Comparison Summary

Summary:

  • You potentially removed 2 lines from the logs
  • Reco comparison results: 67 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 172823
  • DQMHistoTests: Total failures: 8405
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 164418
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 3197.21 KiB( 6 files compared)
  • DQMHistoSizes: changed ( 37634.7503,... ): 1598.605 KiB HLT/SecondaryVertices
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: found differences in 3 / 6 workflows

NVIDIA_T4 Comparison Summary

Summary:

  • You potentially removed 2 lines from the logs
  • Reco comparison results: 151 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 172823
  • DQMHistoTests: Total failures: 14817
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 158006
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 3197.21 KiB( 6 files compared)
  • DQMHistoSizes: changed ( 37634.7503,... ): 1598.605 KiB HLT/SecondaryVertices
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: found differences in 2 / 6 workflows

@fwyzard

fwyzard commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

The duplicate/invalid dictionary errors are caused by #51577 .

@JanGerritSchulz

JanGerritSchulz commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

The duplicate/invalid dictionary errors are caused by #51577 .

Just for my understanding, are these real issues which need to be fixed, or are those pseudo-issues that only happen because #51577 was included on top of the lastest IB instead of being part of an IB directly?
Because e.g. about the duplicated plugins, I moved them into the (previously not-existing) plugins folder. So, I'd expect them to be fully moved (instead of copied) with a clean new IB. Or am I missing something?

@makortel

Copy link
Copy Markdown
Contributor

Because e.g. about the duplicated plugins, I moved them into the (previously not-existing) plugins folder. So, I'd expect them to be fully moved (instead of copied) with a clean new IB.

Right, I'd expect the "duplicate plugins" warning to disappear after the first full build IB (patch build IBs won't be enough).

@mmusich

mmusich commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

@Electricks94 this branch now has developed conflicts to be resolved.

@mmusich

mmusich commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

type ngt

@Electricks94

Copy link
Copy Markdown
Contributor Author

@Electricks94 this branch now has developed conflicts to be resolved.

resolved

@cmsbuild

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51670/50729

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #51670 was updated. @cmsbuild, @fwyzard, @makortel can you please check and sign again.

Comment on lines 10 to +11
#include "DataFormats/SoATemplate/interface/SoALayout.h"
#include "DataFormats/SoATemplate/interface/SoABlocks.h"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#include "DataFormats/SoATemplate/interface/SoALayout.h"
#include "DataFormats/SoATemplate/interface/SoABlocks.h"
#include "DataFormats/SoATemplate/interface/SoABlocks.h"
#include "DataFormats/SoATemplate/interface/SoALayout.h"

@@ -1 +1 @@
#include <iostream>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#include <array>
#include <iostream>

@cmsbuild

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51670/50730

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #51670 was updated. @cmsbuild, @fwyzard, @makortel can you please check and sign again.

@fwyzard

fwyzard commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

enable gpu

@fwyzard

fwyzard commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

please test

@fwyzard

fwyzard commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

+heterogeneous

@cmsbuild

Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs after it passes the integration tests. This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @ftenchini, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Failed Tests: UnitTests
Size: This PR adds an extra 36KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-83c72b/55555/summary.html
COMMIT: 7162e13
CMSSW: CMSSW_20_1_X_2026-08-26-2300/el9_amd64_gcc14
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51670/55555/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 1 errors in the following unit tests:

---> test SoAStreamInternal_t had ERRORS

Comparison Summary

Summary:

  • You potentially added 1 lines to the logs
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 45
  • DQMHistoTests: Total histograms compared: 3592269
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3592251
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
  • Checked 195 log files, 163 edm output root files, 45 DQM output files
  • TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 71 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 8891
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 164848
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 69 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 9345
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 164394
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 102 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 9105
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 164634
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 113 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 9427
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 164312
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 74 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 8528
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 165211
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

@cmsbuild

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51670/50738

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #51670 was updated. @cmsbuild, @fwyzard, @makortel can you please check and sign again.

@fwyzard

fwyzard commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

please test

@fwyzard

fwyzard commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

+heterogeneous

@cmsbuild

Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs after it passes the integration tests. This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @mandrenguyen, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Size: This PR adds an extra 36KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-83c72b/55564/summary.html
COMMIT: b637208
CMSSW: CMSSW_20_1_X_2026-08-26-2300/el9_amd64_gcc14
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51670/55564/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 2 lines to the logs
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 45
  • DQMHistoTests: Total histograms compared: 3592269
  • DQMHistoTests: Total failures: 3
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3592248
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
  • Checked 195 log files, 163 edm output root files, 45 DQM output files
  • TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 95 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 8664
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 165075
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

NVIDIA_H100 Comparison Summary

Summary:

NVIDIA_L40S Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 113 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 9207
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 164532
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 84 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 173739
  • DQMHistoTests: Total failures: 8770
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 164969
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 37634.7503_TTbar_14TeV+Run4D127_HLTHeterogeneousValid step2 max memory diff -38.2 exceeds +/- 30.0 MiB

@ftenchini

Copy link
Copy Markdown
Contributor

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants