Added support for multiple user-specified applications to be run in integration tests by bieryAtFnal · Pull Request #168 · DUNE-DAQ/integrationtest

bieryAtFnal · 2026-08-05T17:32:58Z

Description

In response to a request from Emir for the ability to start a drunc-process-manager instance in addition to the drunc-unified-shell instance in an integration test, functionality has been added to the integrationtest infrastructure to support user-specified applications.

This new functionality is controlled by a new special variable that developers can include in their integtests. This new special variable is described in a new document in this repository. This document will eventually describe all of the special variables that can/should be used in integtests, and it can be viewed here.

These changes have been made in a largely backward-compatible way, so most existing regression/integration tests will not need to be changed. However, the status information that is passed back to the integtest/pytest "tests" now has information about all of the control applications that were run in the DAQ session, so integtests that inspect those results will now need to loop over the list of results instead of simply looking at a single status object.

The console output that users will see when they run an integtest with these changes is largely unchanged, but at a verbosity level of 4 or greater, users will now see information about the processes that are started and the commands that are sent to them.

These changes are correlated with the ones in DUNE-DAQ/drunc#997. The changes in the drunc repo include a new integtest that demonstrates multiple user-specified applications that are run in the DAQ session for the test.

One of the reasons for creating this PR in "draft" mode is that there are still some changes needed in the drunc/process_manager_test.py in order to get it to work.

Here are suggested commands for testing these changes:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260805_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b kbiery/multi_ctrl_proc_support

cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/multi_ctrl_proc_support
git clone https://github.com/DUNE-DAQ/drunc.git -b kbiery/multi_ctrl_proc_support
cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh --verb 4 -r drunc

echo ""
echo -e "\U1F535 \U2705 Note that the new basic_multiapp_test integtest worked, but the process_manager_test did not. \U2705 \U1F535"
echo ""
echo ""

Type of change

New feature or enhancement (non-breaking change which adds functionality)

Testing checklist

Unit tests pass (e.g. dbt-build --unittest)
Minimal system quicktest passes (pytest -s minimal_system_quick_test.py)
Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

Code is commented where needed, particularly in hard-to-understand areas

…to kbiery/multi_ctrl_proc_support

…proc_support

…le_exclusions' into kbiery/multi_ctrl_proc_support

… UTC.

…grationtest_commandline.py (we now have full run control console output stored in log files).

… to help make their meaning and use more clear; also cleaned up the code and comments a little bit.

…cts like filenames, preferring names of applications instead, since they are more durable.

…proc_support

… integtest.

…sed to all apps that support them, not just the unified shell.

…active

…proc_support

…culating parameterization combinations; removed the unneeded call to check_system_resources in create_config_files

…rationtest_drunc.py

…used in integtests.

Updated links and added code snippets for DAQ session ingredients.

…strings to InformationAboutSpecialVariables.md.

…lVariables.md

…proc_support

emmuhamm · 2026-08-06T13:45:43Z

+cmd_set_1 = DAQCommandSet("drunc", dunerc_commands_1, CommandWaitParameters(style=CommandWaitStyle.ECHO))
+cmd_set_2 = DAQCommandSet("pmshell", pmshell_command, CommandWaitParameters(style=CommandWaitStyle.TIME))
+cmd_set_3 = DAQCommandSet("drunc", dunerc_commands_2, CommandWaitParameters(style=CommandWaitStyle.ECHO))


Can you say a few words in the docs about what DAQCommandSet does and what CommandWaitParaameters refer to and their purpose? the command set looks quite self explanatory from the example but I'm still not sure what the CommandWaitParameters are for, especially the difference between ECHO and TIME

I've added some explanation about CommandWaitParameters in the InformationAboutSpecialVariables.md documentation. Please let me know if this helps.

…mpt that appears at the front of an output line.

…y_helper.py.

PawelPlesniak · 2026-08-14T13:11:14Z

I ran the tests with today's nightly, and the tests behaved as expected. The PM passed, but will be followed up with your PR

🔵 Starting test 1 of 11... 🔵
⮕ Running drunc/basic_multiapp_test.py ⬅
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /nfs/home/pplesnia/nightlyDev/NFD_DEV_260814_A9
plugins: anyio-4.12.1, integrationtest-4.0.0
collected 3 items

.venv/lib/python3.12/site-packages/drunc/integtest/basic_multiapp_test.py 

DEBUG: CPU count is 64, required number is 4, recommended number is 8
DEBUG: Free memory is 163.83 GB, required amount is 4, recommended amount is 6
DEBUG: Free disk space on "/tmp" is 19.340557098388672 GB, required amount is 1

----------------------------------------
*** Messages related to configuration generation have been suppressed ***
----------------------------------------


[integtest_proc_mgmt 11:46:17Z] Starting "['drunc-process-manager', 'ssh-standalone', '51957']" with process name "pm"...

[integtest_proc_mgmt 11:46:19Z] Starting "['drunc-process-manager-shell', 'grpc://localhost:51957']" with process name "pmshell"...

[integtest_proc_mgmt 11:46:21Z] Starting "['drunc-unified-shell', 'grpc://localhost:51957', '/tmp/pytest-of-pplesnia/pytest-468/config0/integtest-session-resolved.data.xml', 'smallfootprint', 'smallfootprint']" with process name "drunc"...

[integtest_proc_mgmt 11:46:23Z] Started 3 process(es).

[integtest_proc_mgmt 11:46:25Z] Sent command to pm: help
[integtest_proc_mgmt 11:46:27Z] Sent command to pmshell: help
[integtest_proc_mgmt 11:46:29Z] Sent command to drunc: help
[integtest_proc_mgmt 11:46:31Z] Sent command to drunc: boot
[integtest_proc_mgmt 11:46:31Z] Sent command to drunc: conf
[integtest_proc_mgmt 11:46:31Z] Sent command to drunc: start --run-number 101
[integtest_proc_mgmt 11:46:31Z] Sent command to drunc: wait 1
[integtest_proc_mgmt 11:46:31Z] Sent command to drunc: enable-triggers
[integtest_proc_mgmt 11:46:31Z] Sent command to drunc: wait 20
[integtest_proc_mgmt 11:46:31Z] Sent command to drunc: disable-triggers
[integtest_proc_mgmt 11:46:31Z] Sent command to drunc: echo '*** COMMAND HAS COMPLETED ***'
++++++++++ DAQ Session BEGIN ++++++++++
[drunc] [2026/08/14 11:46:31 UTC] INFO       process_manager_driver.py:123            drunc.process_manager_driver                       Booting session smallfootprint
[drunc] [2026/08/14 11:46:41 UTC] INFO       shell_utils.py:612                       drunc.controller.iface.shell_utils                 Running transition 'conf' on controller 'root-controller', targeting: 'root-controller'
[drunc] [2026/08/14 11:46:41 UTC] INFO       shell_utils.py:612                       drunc.controller.iface.shell_utils                 Running transition 'start' on controller 'root-controller', targeting: 'root-controller'
[drunc] [2026/08/14 11:46:42 UTC] INFO       commands.py:52                           drunc.controller.iface                             Command wait running for 1 seconds.
[drunc] [2026/08/14 11:46:43 UTC] INFO       shell_utils.py:612                       drunc.controller.iface.shell_utils                 Running transition 'enable_triggers' on controller 'root-controller', targeting: 'root-controller'
[drunc] [2026/08/14 11:46:44 UTC] INFO       shell_utils.py:281                       drunc.utils.ShellContext                           Current FSM status is running. Available transitions are disable-triggers, change-rate. Available sequence commands are shutdown, stop-run.
[drunc] [2026/08/14 11:46:44 UTC] INFO       commands.py:52                           drunc.controller.iface                             Command wait running for 20 seconds.
[drunc] [2026/08/14 11:47:04 UTC] INFO       shell_utils.py:612                       drunc.controller.iface.shell_utils                 Running transition 'disable_triggers' on controller 'root-controller', targeting: 'root-controller'
[integtest_proc_mgmt 11:47:04Z] Sent command to pmshell: ps
[integtest_proc_mgmt 11:47:06Z] Sent command to drunc: drain-dataflow
[integtest_proc_mgmt 11:47:06Z] Sent command to drunc: stop-trigger-sources
[integtest_proc_mgmt 11:47:06Z] Sent command to drunc: stop
[integtest_proc_mgmt 11:47:06Z] Sent command to drunc: wait 2
[integtest_proc_mgmt 11:47:06Z] Sent command to drunc: scrap
[integtest_proc_mgmt 11:47:06Z] Sent command to drunc: terminate
[integtest_proc_mgmt 11:47:06Z] Sent command to drunc: echo '*** COMMAND HAS COMPLETED ***'
[drunc] [2026/08/14 11:47:06 UTC] INFO       shell_utils.py:612                       drunc.controller.iface.shell_utils                 Running transition 'drain_dataflow' on controller 'root-controller', targeting: 'root-controller'
[drunc] [2026/08/14 11:47:06 UTC] INFO       shell_utils.py:612                       drunc.controller.iface.shell_utils                 Running transition 'stop_trigger_sources' on controller 'root-controller', targeting: 'root-controller'
[drunc] [2026/08/14 11:47:07 UTC] INFO       shell_utils.py:612                       drunc.controller.iface.shell_utils                 Running transition 'stop' on controller 'root-controller', targeting: 'root-controller'
[drunc] [2026/08/14 11:47:07 UTC] INFO       commands.py:52                           drunc.controller.iface                             Command wait running for 2 seconds.
[drunc] [2026/08/14 11:47:09 UTC] INFO       shell_utils.py:612                       drunc.controller.iface.shell_utils                 Running transition 'scrap' on controller 'root-controller', targeting: 'root-controller'
[drunc] [2026/08/14 11:47:11 UTC] INFO       shell_utils.py:281                       drunc.utils.ShellContext                           Current FSM status is initial. Available transitions are conf. Available sequence commands are start-run.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'mlt' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'dfo-01' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'ru-det-conn-0' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'df-01' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'hsi-01' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'hsi-to-tc-app' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'df-controller' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'ru-controller' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'hsi-controller' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'trg-controller' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[pm] [2026/08/14 11:47:22 UTC] INFO       ssh_process_manager.py:324               drunc.process_manager.SSH_SHELL_process_manager    Process 'root-controller' (session: 'smallfootprint', user: 'pplesnia') was terminated by the process manager through the remote pid. Reported exit code: 0.
[integtest_proc_mgmt 11:47:22Z] Sent command to pm: exit
[integtest_proc_mgmt 11:47:24Z] Sent command to pmshell: exit
[integtest_proc_mgmt 11:47:26Z] Sent command to drunc: exit

The only comments I have are on the definition on some of the variables - I struggled to follow some of them. I will give an abridged discussion below.

For DAQSessionApp - I found this name somewhat confusing. This is currently used to specify the various control applications only. Is it likely to have a different application being deployed using this class? If not, I suggest renaming this to e.g. DAQControlApplication, as we already have a lot of ambiguous use of the term "session".

For CommandWaitParameters and CommandWaitStyle, I struggled to understand the use of these dataclasses, and how they are applied. Could you point me to the relevant information, and include it in the documentation?

Otherwise thank you for this, it looks like it will be very useful.

…proc_support

…ability of the use of the exit command (sent to the apps).

…c_proc_mgmt.py

…roc_mgmt.py, plus a couple nice-to-have changes.

Clarified the explanation of the ECHO wait style and its usage in command sets. Added details about the motivation for the CommandWaitParameters class and its supported wait styles.

Corrected spelling of 'strictly' and improved clarity in the explanation of the 'CommandWaitParameters' class.

added python code type for code snippets

Expanded explanation of CommandWaitParameters and wait styles in integration tests.

bieryAtFnal · 2026-08-18T21:12:38Z

@PawelPlesniak , I've added some text about CommandWaitParameters in this section of the new document. Please let me know if that helps.
I'll look into the variable re-naming that you mentioned.
Thanks

bieryAtFnal · 2026-08-18T21:17:38Z

Here are updated instructions for testing these changes:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260818_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b kbiery/multi_ctrl_proc_support

cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/multi_ctrl_proc_support
git clone https://github.com/DUNE-DAQ/drunc.git -b kbiery/multi_ctrl_proc_support
cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

source $DBT_AREA_ROOT/pythoncode/drunc/scripts/setup_drunc_config_path.sh

dunedaq_integtest_bundle.sh --verb 4 -r drunc

echo ""
echo -e "\U1F535 \U2705 Note that the drunc integtests worked, including the new basic_multiapp_test. \U2705 \U1F535"
echo ""
echo ""

Kurt Biery and others added 27 commits July 26, 2026 14:24

Preliminary changes to support user-specified multiple control apps

6961311

Merge remote-tracking branch 'origin/kbiery/drunc_output_checking' in…

1b36890

…to kbiery/multi_ctrl_proc_support

Merge remote-tracking branch 'origin/develop' into kbiery/multi_ctrl_…

0e30dc9

…proc_support

next round of changes for multi-process support.

2bba24e

Merge remote-tracking branch 'origin/kbiery/updated_drunc_debug_logfi…

9cfeeeb

…le_exclusions' into kbiery/multi_ctrl_proc_support

Added back support for dunerc_options in the async process startups

65c1fce

Modified the times displayed in integtest_proc_mgmt messages to be in…

1b38c71

… UTC.

Removed the obsolete --dunerc-fullprint-watch-string option from inte…

0938bcd

…grationtest_commandline.py (we now have full run control console output stored in log files).

Renamed some vars and data members associated with the intg_proc_mgmt…

c8ec130

… to help make their meaning and use more clear; also cleaned up the code and comments a little bit.

Avoid using process aliases in the intg_proc_mgmt for persistent obje…

32a326d

…cts like filenames, preferring names of applications instead, since they are more durable.

Merge remote-tracking branch 'origin/develop' into kbiery/multi_ctrl_…

736dec1

…proc_support

Fixed problems when multiple confgens or pm types are requested in an…

bcd2910

… integtest.

Modified the handling of run-control-app options so that they are pas…

81acad4

…sed to all apps that support them, not just the unified shell.

Added excluded logfile strings in the PM logfile when drunc-debug is …

c07c4df

…active

Merge remote-tracking branch 'origin/develop' into kbiery/multi_ctrl_…

36fc1f6

…proc_support

Fixed merge errors in log_file_checks.py

45c9f95

Fixed the precedence of the daq_session_ingredients variable when cal…

9b88f9a

…culating parameterization combinations; removed the unneeded call to check_system_resources in create_config_files

Fixed the spelling of parameterizations in several locations in integ…

2f1211c

…rationtest_drunc.py

Added document with information about the special variables that are …

8719efb

…used in integtests.

Added information to the Special Variable document

0fe60d9

Enhance documentation for DAQ session ingredients

7e3a36f

Updated links and added code snippets for DAQ session ingredients.

Added information about placeholder strings in DAQSessionApp startup …

be5248f

…strings to InformationAboutSpecialVariables.md.

Added more detail about placeholder strings in InformationAboutSpecia…

d4e6511

…lVariables.md

Update documentation for DAQSessionIngredients usage

f15cad5

Minor edits

d3c66c1

Fixed basic_multiapp_test.py name

a2bebd6

minor change

da87d7d

This was referenced Aug 5, 2026

Removed the --trigger-full-rc-output option in the dunedaq_integtest_bundle.sh … DUNE-DAQ/daqsystemtest#332

Open

Added an integtest to demonstrate user-specified control applications... DUNE-DAQ/drunc#997

Open

bieryAtFnal requested a review from emmuhamm August 5, 2026 17:53

bieryAtFnal requested review from PawelPlesniak and eflumerf August 5, 2026 17:53

emmuhamm reviewed Aug 6, 2026

View reviewed changes

Comment thread docs/InformationAboutSpecialVariables.md Outdated

Merge remote-tracking branch 'origin/develop' into kbiery/multi_ctrl_…

df87bfd

…proc_support

emmuhamm reviewed Aug 6, 2026

View reviewed changes

Kurt Biery added 2 commits August 6, 2026 10:17

Added logic to async_proc_mgmt::read_stream to remove any command pro…

30b2df1

…mpt that appears at the front of an output line.

Updated the descriptive text for verbosity levels 4 and 5 in verbosit…

10b5710

…y_helper.py.

Kurt Biery and others added 10 commits August 16, 2026 17:50

Merge remote-tracking branch 'origin/develop' into kbiery/multi_ctrl_…

6b9e213

…proc_support

added a TIME_PLUS_EXIT command completion type to try to improve reli…

274afda

…ability of the use of the exit command (sent to the apps).

Fixed the printout of the commands when an exit command fails in asyn…

4b1336a

…c_proc_mgmt.py

Added checks for supported commands in various wait styles in async_p…

74ba70d

…roc_mgmt.py, plus a couple nice-to-have changes.

added information about CommandWaitParameters

641a88b

Enhance documentation on CommandWaitParameters and wait styles

b9616e3

Clarified the explanation of the ECHO wait style and its usage in command sets. Added details about the motivation for the CommandWaitParameters class and its supported wait styles.

Updated SpecialVariables doc

a631758

Fix typos and enhance clarity in documentation

a5918ab

Corrected spelling of 'strictly' and improved clarity in the explanation of the 'CommandWaitParameters' class.

Improve formatting and clarity in documentation

597b2a6

added python code type for code snippets

Update InformationAboutSpecialVariables.md

85c6639

Expanded explanation of CommandWaitParameters and wait styles in integration tests.

bieryAtFnal marked this pull request as ready for review August 18, 2026 21:13

Navigation Menu

Uh oh!

Added support for multiple user-specified applications to be run in integration tests - #168

Added support for multiple user-specified applications to be run in integration tests#168
bieryAtFnal wants to merge 40 commits into
developfrom
kbiery/multi_ctrl_proc_support

bieryAtFnal commented Aug 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

emmuhamm Aug 6, 2026

Uh oh!

bieryAtFnal Aug 18, 2026

Uh oh!

PawelPlesniak commented Aug 14, 2026 •

edited

Loading

Uh oh!

bieryAtFnal commented Aug 18, 2026

Uh oh!

bieryAtFnal commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

bieryAtFnal commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Testing checklist

Further checks

Uh oh!

Uh oh!

emmuhamm Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

bieryAtFnal Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

PawelPlesniak commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bieryAtFnal commented Aug 18, 2026

Uh oh!

bieryAtFnal commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bieryAtFnal commented Aug 5, 2026 •

edited

Loading

PawelPlesniak commented Aug 14, 2026 •

edited

Loading