Handling exceptions when applications do not report errors but time out in stateful transitions - #868

Merged

PawelPlesniak merged 103 commits into

PawelPlesniak/IncompleteStatefulCommandTransition

Aug 6, 2026

Commits on Mar 31, 2026

Generating an environment for which the issue can be recreated
PawelPlesniak
committed
a7ba095
WIP
PawelPlesniak
committed
c50fb9f

Commits on Jun 4, 2026

Merging
PawelPlesniak
committed
db1574e
Resovling duplicate config
PawelPlesniak
committed
456a461
Adding drunc to the DUNEDAQ_DB_PATH (should get replaced but subject to agreement with SWIT as to where to host these configs)
PawelPlesniak
committed
785e98b
Splitting out the failure mode testing into a separate file
PawelPlesniak
committed
7da158d
Making the configs actually pass oks_dump with the DB path
PawelPlesniak
committed
f6f8c62
Remmoving duplicate confs
PawelPlesniak
committed
8859c4b
Constructed tests for death on boot and post boot, as well as long FSM timeouts and death on FSM, REQUIRES TESTING
PawelPlesniak
committed
234f185
Adding death cases
PawelPlesniak
committed
1bf07c6

Commits on Jun 5, 2026

Clearer conf param names
PawelPlesniak
committed
34d45b9
Configs developed, time to test
PawelPlesniak
committed
c4ead7f
Remove dupe var definitions
PawelPlesniak
committed
cf4ddb8
Fixing the broken session -> OKS var links
PawelPlesniak
committed
2a9348a
Included the configs, damned be gitignore
PawelPlesniak
committed
c8fa56e
Vars rechecked
PawelPlesniak
committed
1323cce
Moving point of timeout
PawelPlesniak
committed
abde856
WIP
PawelPlesniak
committed
468bef9
Root controller is now marked as in error on timeout, but this needs to be propagated down to the actual failing app level
PawelPlesniak
committed
92a5314

Commits on Jun 8, 2026

Printing the status after marking an error state
PawelPlesniak
committed
1aa6d05
Adding log RPC
PawelPlesniak
committed
2917d12
Logs on remote server functional
PawelPlesniak
committed
899c510
WIP
PawelPlesniak
committed
6554774

Commits on Jun 9, 2026

WIP continuned
PawelPlesniak
committed
b169b22
Logfiles now have a proper format
PawelPlesniak
committed
57a8cfa
Docs part 1
PawelPlesniak
committed
0b22161

Commits on Jun 10, 2026

Argparse -> click
PawelPlesniak
committed
ba65585
Constructing integ test, renaming test dir to follow convention
PawelPlesniak
committed
6451ede
Integration tests almost running
PawelPlesniak
committed
f0514d0
Silly circular dependency
PawelPlesniak
committed
e4f82a1

Commits on Jun 11, 2026

First test working
PawelPlesniak
committed
cc9de6e

Commits on Jun 15, 2026

Test renamed
PawelPlesniak
committed
cc26c52
Renaming for clearer implementation, not working as of this commit
PawelPlesniak
committed
f929927
Adding 'no stop on error in batch mode' option to the unfiied shell
PawelPlesniak
committed
b59e4af
Why do multiple tests nopt work!
PawelPlesniak
committed
3dd7bd5
Once again, bare functionality of the deat on boot nest app config
PawelPlesniak
committed
9513d86
Next test working
PawelPlesniak
committed
4d6ac3f
Adding both integ tests to the integtest_list
PawelPlesniak
committed
305a89e
Disabling coverage reporting if integration tests are ran
PawelPlesniak
committed
e5d91d8
Commas suck and .sh scripts hate them
PawelPlesniak
committed
50a6335
Updating the relative paths
PawelPlesniak
committed
fb0e98a

Commits on Jun 16, 2026

Docs and getting the applications to succesfully die post boot
PawelPlesniak
committed
b3adf82
Post boot test WIP
PawelPlesniak
committed
72bd6e6
Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTransition
PawelPlesniak
committed
14898e5
Session is FINALLY getting the top controller name correct, and it is reproducible.
PawelPlesniak
committed
d508f2f

Commits on Jun 17, 2026

App dying DOES NOT GET MARKED AS DEAD, but gets put into a disconnected state and the session gets put into error on boot
PawelPlesniak
committed
b3e7e61
Apps correctly dyign when told to
PawelPlesniak
committed
cb101b0
Adding signal handling to the fake_daq_app
PawelPlesniak
committed
2d31ed9
Cleanup
PawelPlesniak
committed
0238d96
Ensuring the sessions do not even attempt to execute stateful commands when the session is in error and when in batch mode
PawelPlesniak
committed
5d8ff8a
Placing session in error state is failing badly
PawelPlesniak
committed
240dc13
Next round of test complete
PawelPlesniak
committed
ac04be0
WIP
PawelPlesniak
committed
e7b9907
FSM test behaves differently with integration test use vs manual batch mode use, and it is very wierd
PawelPlesniak
committed
f6b8265
Another broken test, to see if it was test speciffic
PawelPlesniak
committed
7ca462a

Commits on Jul 8, 2026

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTransition
PawelPlesniak
committed
2f0e902
Disabling pytest if a PR is in draft mode
PawelPlesniak
committed
f7457e8
Small modernize
PawelPlesniak
committed
4301b28

Commits on Jul 9, 2026

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTransition
PawelPlesniak
committed
82f8ed0
WIP
PawelPlesniak
committed
23f72c9
First test should be working again, the remainder should be parsed quickly
PawelPlesniak
committed
960f3ca

Commits on Jul 10, 2026

deat on boot nest app done
PawelPlesniak
committed
8886718
Only missing one test, then need to do a LOT of documentation
PawelPlesniak
committed
a2a418b
Needs documentation
PawelPlesniak
committed
2d22271

Commits on Jul 13, 2026

WIP
PawelPlesniak
committed
91b1e97
Removing the reference config
PawelPlesniak
committed
3d8c5b7
Pytest fix
PawelPlesniak
committed
0da7956

Commits on Jul 16, 2026

log_on_server WIP
PawelPlesniak
committed
9c54136
Done?
PawelPlesniak
committed
1b6e2c0
Reducing verboisty of an error message
PawelPlesniak
committed
8543da7
Small fixes
PawelPlesniak
committed
b430dd1
Updating the driver script for integration tests
PawelPlesniak
committed
67f39c4

Commits on Jul 17, 2026

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTransition
PawelPlesniak
committed
2e2dcab
Merge complete
PawelPlesniak
committed
bcf5fe7
Ruff linting
PawelPlesniak
committed
c9a9cc4

Commits on Jul 21, 2026

Commits on Jul 23, 2026

Clean up minor issues from other tests
emmuhamm
committed
99149a6
Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTransition
emmuhamm
committed
ee14f62
Fix failures
emmuhamm
committed
15d1735

Commits on Jul 24, 2026

Update src/drunc/integtest/test_failure_mode_death_on_boot_nest_app.py

PawelPlesniak
and
emmuhamm
authored
113b82b
Adding requested feature to state what command to execute when an application times out
PawelPlesniak
committed
215f0c8
extending tests
PawelPlesniak
committed
c6717ab
Merge branch 'PawelPlesniak/IncompleteStatefulCommandTransition' of github.com:DUNE-DAQ/drunc into PawelPlesniak/IncompleteStatefulCommandTransition
PawelPlesniak
committed
291e6cc

Commits on Aug 3, 2026

Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTransition
PawelPlesniak
committed
f23c153
Centralizing the DB path check logic
PawelPlesniak
committed
5e91435
Adding the relevant docstrings
PawelPlesniak
committed
125bd59
Renaming the tests for naming consistency
PawelPlesniak
committed
f1df46d
Documentation and comment addressing
PawelPlesniak
committed
4ebe2d0
Making it clearer which application the user should focus on when running these tests
PawelPlesniak
committed
f3195c3
Update src/drunc/integtest/failure_mode_death_on_boot_nest_app_test.py

PawelPlesniak
and
emmuhamm
authored
e7ee9a7
Simplifying the pytest
PawelPlesniak
committed
2f14250
Changing dependency from external to internal
PawelPlesniak
committed
ae81235
Merge branch 'PawelPlesniak/IncompleteStatefulCommandTransition' of github.com:DUNE-DAQ/drunc into PawelPlesniak/IncompleteStatefulCommandTransition
PawelPlesniak
committed
19cadc1
Simplifying logic
PawelPlesniak
committed
745f5af

Commits on Aug 4, 2026

Minor updates to logging and testing output
PawelPlesniak
committed
e6ad1ab
Correcting pytests, failures associated with bad merges, and resolving another issue with this PR
PawelPlesniak
committed
45cba8d
Changing excluded nodes to be included
PawelPlesniak
committed
88fb003
Merge branch 'develop' into PawelPlesniak/IncompleteStatefulCommandTransition
PawelPlesniak
committed
c4fa1f2
Remvoing resisudal time.sleep
PawelPlesniak
committed
5fc0a4d
Genearlizing the drunc DUNEDAQ_DB_PATH check, and fixing the ruff config
PawelPlesniak
committed
646bade
Adding a ternary preference
PawelPlesniak
committed
4535848

Commits on Aug 5, 2026

Adding a clearer comment
PawelPlesniak
committed
2050f3c