Skip to content

In integration tests, store drunc console output in a log file and check for warnings and errors #162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bieryAtFnal merged 8 commits into from
Jul 27, 2026
Merged

In integration tests, store drunc console output in a log file and check for warnings and errors #162

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
73113a4
Added functionality to integrationtest_drunc.py to store the drunc co…
Jul 15, 2026
4f55e76
Added a phrase that we always want to exclude from drunc log file che…
Jul 15, 2026
7e8cab7
Fixed a bug in log_file_checks.py - only the first excluded or requir…
Jul 15, 2026
4cc6708
added critical messages to ones that we'll print out from the drunc c…
Jul 21, 2026
f0e3ea4
Updated the standard excluded phrases in log_file_checks.py to includ…
Jul 21, 2026
59a351f
Merge remote-tracking branch 'origin/develop' into kbiery/drunc_outpu…
Jul 22, 2026
a7716ec
Filtered out ANSI escape characters in the text that we write to the …
Jul 26, 2026
e2e2450
Dropped the 'X11 forwarding' message from the excluded list in log_fi…
Jul 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions src/integrationtest/integrationtest_drunc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -613,8 +613,8 @@ class RunResult:
cwd=run_dir
)

# print out each line of captured output, as well as add it to the string that we
# pass back to the user, subject to the verbosity level that the user has requested
# print out each line of captured output, subject to the verbosity level that the
# user has requested, as well as add it to the string that we pass back to the user
tmp_string = request.config.getoption("--dunerc-fullprint-watch-string")
full_printout_watch_string = tmp_string.replace("_SPC_", " ")
full_printout_activated = False
Expand Down Expand Up @@ -643,7 +643,9 @@ class RunResult:

# check for errors and warnings for all verbosity levels
if should_be_printed == False:
if ("error" in line.lower() and (not "In error" in line and not "Endpoint" in line)) or "warning" in line.lower():
lc_line = line.lower()
if ("error" in lc_line and (not "In error" in line and not "Endpoint" in line)) \
or "warning" in lc_line or "critical" in lc_line:
should_be_printed = True

# check for basic transition messages, if that level of verbosity is requested
Expand Down Expand Up @@ -673,6 +675,10 @@ class RunResult:
rc_process.communicate()
proc_returncode = rc_process.returncode

# store the full dunerc console output in a log file for reference and checking
with open(f"{run_dir}/log_{getpass.getuser()}_drunc_console_output.txt", "w", encoding="utf-8") as ff:

@PawelPlesniak PawelPlesniak Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is exactly what is needed, but I suggest an extension here. Investigating the logfile, mani ANSI escape characters associated with the use of the rich output with color coding is including additional verbosity as

�[2;34m[2026/07/24 11:47:01 UTC]�[0m �[1;32mINFO      �[0m �[2;37mshell.py:158                            �[0m �[2;37mdrunc.unified_shell                               �[0m User pplesnia �[32mstarting the unified_shell�[0m
�[2;34m[2026/07/24 11:47:01 UTC]�[0m �[1;32mINFO      �[0m �[2;37mshell.py:214                            �[0m �[2;37mdrunc.unified_shell                               �[0m �[32mSetting up the SSH_SHELL process manager�[0m with configuration �[32mssh-standalone�[0m
�[2;34m[2026/07/24 11:47:01 UTC]�[0m �[1;32mINFO      �[0m �[2;37mshell.py:231                            �[0m �[2;37mdrunc.unified_shell                               �[0m Starting process manager
�[2;34m[2026/07/24 11:47:01 UTC]�[0m �[1;32mINFO      �[0m �[2;37mprocess_manager.py:111                  �[0m �[2;37mdrunc.process_manager                             �[0m process_manager communicating through address �[1;92m10.73.136.70�[0m�[1;32m:�[0m�[1;36m38319�[0m
�[2;34m[2026/07/24 11:47:01 UTC]�[0m �[1;32mINFO      �[0m �[2;37mshell.py:291                            �[0m �[2;37mdrunc.unified_shell                               �[0m Setting up the controller interface

There was a similar case of this when @emmuhamm overhauled the python logging framework, and the stripping of this was implemented in daqsystemtest in this PR. Would it be possible to implement this?

ff.write(full_output)

# construct a CompletedProcess instance to be passed back to the user. In this way,
# user code does not need to change in response to the change in this code from
# using subprocess.run() to subprocess.Popen().
Expand Down
9 changes: 6 additions & 3 deletions src/integrationtest/log_file_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ def log_has_no_errors(log_file_name, print_logfilename_for_problems=True, exclud
severity=match_logline_prefix.group(1)
if severity in ("WARNING", "ERROR", "FATAL"):
bad_line=True
#if not bad_line:

@PawelPlesniak PawelPlesniak Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be removed?

else: # This line's not produced with our logging package, so let's just look for bad words
if "WARN" in line or "Warn" in line or "warn" in line or \
"ERROR" in line or "Error" in line or "error" in line or \
Expand Down Expand Up @@ -111,6 +112,10 @@ def logs_are_error_free(log_file_names, show_all_problems=True, print_logfilenam
["RAN:", "LogLevel=error"]
)

# 21-Jul-2026, KAB: phrases that we always want to exclude
excluded_substring_map.setdefault("drunc", []).extend(["Substate.*In error.*Endpoint"])

@PawelPlesniak PawelPlesniak Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the movtivation for excluding the phrase "Endpoint" or "Substate" here?

excluded_substring_map.setdefault(r".*", []).extend(["X11 forwarding setup failed"])

@PawelPlesniak PawelPlesniak Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This issue should have been addressed by DUNE-DAQ/drunc#982, I would suggest removing it so we can keep track of whether this issue re-appears in the future


all_ok=True
#print("") # Clear potential dot from pytest
for log in log_file_names:
Expand All @@ -121,13 +126,11 @@ def logs_are_error_free(log_file_names, show_all_problems=True, print_logfilenam
match_obj = re.search(exclusion_key, log.name)
if match_obj:
exclusions += excluded_substring_map[exclusion_key]
break
for required_key in required_substring_map.keys():
match_obj = re.search(required_key, log.name)
if match_obj:
requireds += required_substring_map[required_key]
break


single_ok=log_has_no_errors(log, print_logfilename_for_problems, exclusions, requireds,
print_required_message_report, verbosity_helper)

Expand Down