Add split shell PMaaS integtests by emmuhamm · Pull Request #1001 · DUNE-DAQ/drunc

emmuhamm · 2026-08-09T12:53:18Z

Description

Builds on #997

New tests

Adds three tests:

pm_pms
- Checks to see if basic pms works
- Checks for all the relevant process manager commands (basically ports process_manager_tests to pmaas)
- Checks for logs on both pm and pms
- Checks for connections
- Checks if its possible to boot multiple processes
- Does not check what happens if you try to boot the same session if its still up
pm_us
- Checks to see if basic US works
- Only does boot and terminate so far, as controller commands are still uncertain
pmaas
- Cycles pmaas default checks
- Booting multiple sessions in pms, and booting in us, on the same pm
- Checks if logs work on both pms/us
- Checks if terminate on us means the other sessions remain up

Relevant developments

Several developments need to happen in order for this to work

added echo to the process manager
added echo-on-server to the process manager
- Functions similarly to log-on-server, except it logs to drunc.echo instead of self.log
added echo-on-server to the unified shell as well
- Purposely did not touch controller code, so there is a warning if echo-on-server is called for the controller

Dependencies

Some dependencies as well

Kurt's work, naturally (Added an integtest to demonstrate user-specified control applications... #997)
emmuhamm/echo-on-server druncschema (No PR yet)

Testing

Important note: This is currently targetting kurt's branch, which is out of date with develop. To test this, this branch will need to be build on top of kurt's branch, which has to also be built on top of an up to date develop.

There is a branch with the right dependencies in drunc: emmuhamm/split-shell-tests-correct-dependencies to test if the thingys work
Need the dependency on druncschema as well
Run pytest -s --tb=short /nfs/home/emmuhamm/nightly/NFD_DEV_260807_A9/pythoncode/drunc/src/drunc/integtest/pm*.py --integtest-verbosity 5
Or alternatively, run the usual drunc integtest bundle

Tested on np04-srv-028 on 07 aug 26 nightly, all integtests (except failure_.py) passes. Haven't testred on failure_.py due to issues reported by Kurt

Ruff and pytest and msqt and stuff haven't been done since still waiting on Kurts stuff. MSQT passes though

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: WHAT HOSTNAME from release RELEASE_NAME

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

Unit tests (pytest --marker) passed
- With relevant marker _INSERT MAKER NAME HERE_
- Relying on the CI workflow
Integration tests passed
- Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
- Full daqsystemtest_integtest_bundle.sh
Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows
Drunc integration tests pass (dunedaq_integtest_bundle.sh -r drunc)

Final checklist prior to marking this as "Ready for Review"

Code is clearly commented.
New unit tests have been added, or is documented in # ISSUE NUMBER
A suitable reviewer has been chosen from this list.

Reviewer checklist

This branch has been rebased with develop prior to testing.
Suggested manual tests show changes.
CI workflows fails documented (if present)
Integration tests passed (on either np0x or IC HEP clusters)
- Use the following guidelines to determine which of the integration tests you need to run
  - You do not need to run any integration tests if
    - Code changes are not associated with src/
    - PR changes only affect docstrings
    - In this case, be sure to validate any suggested manual testing.
  - Run only the minimum integration test as daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py if
    - PR changes only affect a few log entries
    - PR changes are small, and do not have a large impact on the workflow (use carefully)
  - Otherwise run the full integration test bundle as daqsystemtest_integtest_bundle.sh
- What to do if the integration tests fail?
  - Only concern yourself if failures related to drunc are in the log files
  - If non-drunc failure appears:
    - Validate failure in fresh working area
    - Contact Pawel if unsure
If you have ran the full integration test bundle, leave a comment on the PR stating
- Which host the integration tests have ran on
- [Optional] A copy of the test summary
Drunc integration tests pass (scripts/drunc_integtest_bundle.sh)

Once the above boxes are checked, the PR(s) can be merged following the steps below.

Prior to merging

Choose one of the following an complete all substeps

Changes only affect the Run Control, are in a single repository, and do not affect the end user.
- Changes are documented in docstrings and code comments
- Wiki has been updated if architectural or endpoint changes
Otherwise
- Workflow changes demonstrated in the Change Log (if necessary)
- Wiki has been updated (if necessary)
- #dunedaq-integration Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

emmuhamm · 2026-08-09T13:07:31Z

Ooh stacks exist! This is a feature I did not know I want, but I've been working this way for a loongg time. Lets try it here :)

emmuhamm

Hi @PawelPlesniak, this is ready for a look on your end now. I wouldn't say review since it's still waiting on Kurt's developments, but hey we can build on it already :)

I've left a few comments on things that I thought about and didnt have enough time, or points of interests as usual.

However, in terms of the future I would say that these set of tests is enough to test PMaaS for now, and should be fine to close the PMaaS issue. Of course, we can expand these in the testing framework later on but this might be good for now.

Let me know what you think, and feel free to tag Kurt as well if you think this is mature enough.

emmuhamm · 2026-08-09T14:26:24Z

+# 05-Aug-2026, KAB: the goal of this test is to validate and demonstrate the use of multiple
+# user-specified applications running in the DAQ session that is part of this test.
+#
+# This integtest was created by copying the small_footprint_quick_test from the daqsystemtest
+# repo and converting the assignment of the run control commands to make use of the new
+# "daq_session_ingredients" special integtest variable.


Ah forgot to update the top of the files but I've run out of time :/

emmuhamm · 2026-08-09T14:27:53Z

There should be an issue tackled on cleaning up integ_test_utils, I think its gonna get a little bit webby and tangly

emmuhamm · 2026-08-09T14:29:27Z

+    echo pre_boot_2
+    echo-on-server pre_boot_2
+    ps -w 180
+    boot config/daqsystemtest/example-configs.data.xml local-1x1-config {daq_session_name_1}


as mentioned in the description, we are currently unable to do another boot config/daqsystemtest/example-configs.data.xml local-1x1-config {daq_session_name} because when you do this it comes out with a 'theres another session already booted, are you sure? Y/N`.

Haven't had the time to figure out what to do about this yet so I've left it

emmuhamm · 2026-08-09T14:30:21Z

+    pm_connect = "connected from process_manager_shell"
+    pms_connect = "connected to the process manager through a"


Bit of a broad test, maybe if we want to we can also do a 'check if we have the right user' and 'check if its connecting to the right places (ports) and stuff.

emmuhamm · 2026-08-09T14:30:50Z

+    assert_rows_have_valid_uuids(ps_post_boot)
+
+
+def test_boot_pm(run_dunerc) -> None:


Maybe combine the test_boots into one test?

emmuhamm · 2026-08-09T14:31:25Z

+    )
+
+
+def test_restart_mlt_logs_pms(run_dunerc) -> None:


very light test but does the job, but I wonder if theres other things we can add on this test?

emmuhamm · 2026-08-09T14:31:49Z

+    # TODO: This bit here is grabbing functions from the integ test utils. Maybe it can be better optimised?
+    table_start_idx = find_line_index(
+        lines_pms,
+        lambda line: "Terminated process" in line,
+        start_idx=pre_boot_idx_pms + 1,
+    )
+
+    assert table_start_idx is not None, "cannot fine terminated process table"
+
+    terminated_table = _parse_table_from_index(lines_pm, table_start_idx, _PS_COLUMNS)
+    for row in terminated_table:
+        assert UUID_RE.match(row["uuid"]), (
+            f"Expected a valid UUID for process '{row['friendly_name']}', got '{row['uuid']}'"
+        )


See the comment here. Maybe this would be better left as a function in integ_test_utils?

emmuhamm · 2026-08-09T14:33:04Z

+pm_port = find_free_port(50020, 52000)
+
+# The command lines that should be used to start the applications
+procmsg_startup_commands = ["drunc-process-manager", "<proc_mgr_choice>", str(pm_port)]


Something to keep in mind that so far all of this is near hardcoded to use the ssh-standalone thing.

We should absolutely make tests for cern kafka as well and later on k8s. maybe in the next PR? and maybe this is beyond the scope of PMaaS but something for the testing framework?

emmuhamm force-pushed the emmuhamm/split-shell-tests branch from 5a9fa23 to 5271327 Compare August 9, 2026 13:03

emmuhamm linked an issue Aug 9, 2026 that may be closed by this pull request

Write tests for the split shell functionality #907

Open

emmuhamm changed the title ~~Emmuhamm/split shell tests~~ Add split shell PMaaS integtests Aug 9, 2026

emmuhamm self-assigned this Aug 9, 2026

emmuhamm added 8 commits August 9, 2026 16:23

add echo command to PM

999c943

patch failing ps in pms when no session

ec5d71a

Modify log to use echo in PM

517c663

update restart log

8026a13

Add initial version of tests

4b5914c

Cleanup tests to avoid repetition

d7917f9

Add echo_on_server

9b849e4

patch test to use echo_on_server

cb80f08

emmuhamm force-pushed the emmuhamm/split-shell-tests branch from 67fe207 to cb80f08 Compare August 9, 2026 14:25

emmuhamm commented Aug 9, 2026

View reviewed changes

emmuhamm added Process Manager Testing labels Aug 9, 2026

emmuhamm marked this pull request as ready for review August 9, 2026 14:36

Navigation Menu

Uh oh!

Add split shell PMaaS integtests - #1001

Add split shell PMaaS integtests#1001
emmuhamm wants to merge 8 commits into
kbiery/multi_ctrl_proc_supportfrom
emmuhamm/split-shell-tests

emmuhamm commented Aug 9, 2026 •

edited

Loading

Uh oh!

emmuhamm commented Aug 9, 2026

Uh oh!

emmuhamm left a comment

Uh oh!

emmuhamm Aug 9, 2026

Uh oh!

emmuhamm Aug 9, 2026

Uh oh!

emmuhamm Aug 9, 2026

Uh oh!

emmuhamm Aug 9, 2026

Uh oh!

emmuhamm Aug 9, 2026

Uh oh!

emmuhamm Aug 9, 2026

Uh oh!

emmuhamm Aug 9, 2026

Uh oh!

emmuhamm Aug 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		pm_connect = "connected from process_manager_shell"
		pms_connect = "connected to the process manager through a"

		assert_rows_have_valid_uuids(ps_post_boot)


		def test_boot_pm(run_dunerc) -> None:

Uh oh!

Conversation

emmuhamm commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

New tests

Relevant developments

Dependencies

Testing

Developer checklist

Prior to marking this as "Ready for Review"

Final checklist prior to marking this as "Ready for Review"

Reviewer checklist

Prior to merging

Notification message for a Slack channel

For an single merge that changes the user workflow

For co-ordinated merge

Uh oh!

emmuhamm commented Aug 9, 2026

Uh oh!

emmuhamm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

emmuhamm commented Aug 9, 2026 •

edited

Loading