Skip to content

The 4.14 wazuh-agent and wazuh-manager images have no diff binary, so FIM report_changes never reports what changed #2771

Description

@Rebits
Wazuh version Component Install type Install method Platform
4.14.10-rc1 (also 4.14.9) wazuh/wazuh-agent and wazuh/wazuh-manager images, FIM report_changes Agent and Manager Docker (Docker Hub 4.14.10-rc1) Amazon Linux 2023 base image

Description

FIM builds the content diff of report_changes by running the external diff command (src/syscheckd/src/fim_diff_changes.c, system("diff ...")). Neither 4.14 image installs diffutils, so every change in a directory with report_changes="yes" logs (6714): Command diff output an error and the alert has no syscheck.diff. The FIM alert itself is still produced; only the "what changed" part is lost.

#2594 reported this for 5.0.0-latest and #2620 fixed it on 5.0.0, in the agent image only. The 4.14.10 and 4.14.11 branches still install no diffutils:

  • build-docker-images/wazuh-agent/Dockerfile:19: yum install curl-minimal tar gzip procps -y
  • build-docker-images/wazuh-manager/Dockerfile:16: yum install curl-minimal xz gnupg tar gzip openssl findutils procps -y

The manager image is affected as well because in 4.x wazuh-syscheckd also runs on the manager.

Image command -v diff rpm -q diffutils Digest
wazuh/wazuh-agent:4.14.9 not found not installed sha256:db274c1e2c56106f108fa1469f7ee99995ad3cac41003e8ca7a975cda2af20d8
wazuh/wazuh-agent:4.14.10-rc1 not found not installed sha256:4d307cbba42a5f50aefb514c65722d16fba702507a803354423742a0f79cb923
wazuh/wazuh-manager:4.14.9 not found not installed sha256:dbb337727a6b6b17e24b1ccfe4c6e95fd983286ba0ccbf29d568144aa9527c6b
wazuh/wazuh-manager:4.14.10-rc1 not found not installed sha256:6eee970aa8d53f89770b801bd09b86d9d462bfcabbecb39f48c608fa5ee738aa

Steps to reproduce

  1. Run the single-node stack and a wazuh/wazuh-agent:4.14.10-rc1 container enrolled to it.
  2. Monitor a directory with content diffs, on the agent (or in the manager's ossec.conf):
    <directories realtime="yes" report_changes="yes" check_all="yes">/fimtest</directories>
  3. Create /fimtest/keep.txt, wait for the baseline scan, then append a line: echo marker >> /fimtest/keep.txt.
  4. Check /var/ossec/logs/ossec.log in the container and the syscheck.diff field of the rule 550 alert.

Expected behavior

The rule 550 alert carries syscheck.diff with the appended line, as it does on a host that has diff.

Current behavior

Same marker appended on four hosts at 11:04:55 UTC. ubuntu24 (Ubuntu 24.04 with the wazuh-agent_4.14.10-1_amd64.deb pre-release package, which has diff) is the control:

agent-al2  (wazuh/wazuh-agent:4.14.10-rc1)  alert=550 size 18->38  syscheck.diff=null
           ossec.log: 2026/10/26 11:04:55 wazuh-syscheckd: ERROR: (6714): Command diff output an error
ubuntu24   (deb, control)                   alert=550 size 18->38  syscheck.diff="2a3\n> rc1-diff-1793012695\n"
manager    (wazuh/wazuh-manager:4.14.10-rc1) alert=550 size 5->25  syscheck.diff=null
           ossec.log: 2026/10/26 11:04:55 wazuh-syscheckd: ERROR: (6714): Command diff output an error

Control: after dnf -y install diffutils in the agent-al2 container only, with no restart, the next change (11:05:54 UTC) has a diff on agent-al2 while the manager still has none:

agent-al2  alert=550 size 38->58  syscheck.diff="2a3,4\n> rc1-diff-1793012695\n> rc1-diff-1793012754\n"
manager    alert=550 size 25->45  syscheck.diff=null
           ossec.log: 2026/10/26 11:05:54 wazuh-syscheckd: ERROR: (6714): Command diff output an error

The attached screenshot shows the same 8 alerts in Discover.

Proposed fix

Backport #2620 to the 4.14.x branches and add diffutils to the manager image too. The wazuh-agent and wazuh-manager RPMs declare no dependency on diffutils (only coreutils), so the same error appears on a CentOS Stream 9 container with the RPM installed; a Requires: diffutils in the specs would also cover the images, which install those RPMs.

Activity

  1. moved this from Backlog to In progress in XDR+SIEM/Release 4.14.11on Nov 2, 2026
  2. vcerenu commented on Nov 2, 2026

    @vcerenu
    Member

    Dependencies the images need for the default configuration

    The fix for this issue only adds diffutils. While checking it, every service in the default single-node stack plus an agent was traced, on 4.14.10-rc1 and 5.0.0-latest. The trace recorded every execve() and every shell command not found.

    The commands below are missing from the images. Gaps that only affect the manager's monitoring of its own container are left out, and so are the Wazuh indexer and the Wazuh dashboard, which need nothing.

    This comment is a heads-up; nothing beyond diffutils is being changed here.

    1. Default agent features that need a missing command

    Command AL2023 package Used by 4.14 agent 5.0 agent Effect
    diff diffutils FIM report_changes (fim_diff_changes.c:747,759, system("diff …")) missing missing (6714): Command diff output an error, no diff in the alert or event. Fixed by this issue
    netstat net-tools <localfile> full_command netstat -tulpn | … missing n/a empty output: rule 533 (listening ports changed) never fires
    last util-linux <localfile> full_command last -n 20 missing n/a empty output: rule 535 (last logged-in users) never fires

    In 4.14, the two <localfile> blocks are in the ossec.conf packaged in the agent image (lines 171 and 178) and in wazuh-agent/config/wazuh-agent-conf:164,171. The 5.0.0 agent configuration has no command <localfile> blocks.

    Trace from the 4.14.10-rc1 agent:

    sh: line 1: diff: command not found       <- /var/ossec/bin/wazuh-syscheckd
    sh: line 1: last: command not found       <- /var/ossec/bin/wazuh-logcollector
    sh: line 1: netstat: command not found    <- /var/ossec/bin/wazuh-logcollector
    sh: line 1: xargs: command not found      <- /var/ossec/bin/wazuh-modulesd (SCA)
    

    2. SCA, enabled by default on the agent

    The agent evaluates cis_amazon_linux_2023.yml (183 checks in 4.14, 182 in 5.0). These commands are called by the policy and are not in the images, so those checks are evaluated without the command's output.

    Command AL2023 package Checks that call it 4.14 agent 5.0 agent
    findmnt util-linux-core 26 missing missing
    sshd openssh-server 16 missing missing
    auditctl audit 15 not reached missing
    systemctl systemd 11 missing missing
    lsmod, modprobe kmod 3 + 3 missing missing
    nft nftables 3 not reached missing
    getenforce / sestatus libselinux-utils / policycoreutils 2 / 1 missing missing
    augenrules audit 1 missing missing
    crontab cronie 1 missing missing
    passwd passwd 1 missing missing
    ss iproute 1 missing missing
    sudo sudo 1 missing missing
    xargs findutils 1 missing present
    useradd shadow-utils 1 present missing (removed after creating the user)

    "Not reached" means the trace did not get to that call on 4.14, not that the command is installed.

    The SCA policy evaluates the system the agent runs on, here the container. Several of these checks look at host-level services (sshd, systemctl, kernel modules, SELinux, auditd) that an agent container does not run, even if the tools are installed. Installing them would change the SCA results, not make the checks meaningful.

    3. Active response, not bound to any rule by default

    The manager defines the commands and the agent has <active-response> enabled, but no rule triggers them out of the box. If a user enables them, the agent image has none of the binaries they call:

    Response Calls 4.14 agent 5.0 agent
    firewall-drop iptables / ip6tables (firewall-cmd for firewalld-drop) missing missing
    route-null ip / route missing missing
    disable-account passwd missing missing

    The agent Compose files add no cap_add, and Docker does not grant CAP_NET_ADMIN by default, so blocking traffic from an agent container needs more than the packages.

    Not included

    • 4.14 manager: diff, netstat and last are missing as well, but only for the manager's FIM and <localfile> blocks on its own container. The 4.14 fix still adds diffutils there, as this issue asks.
    • 5.0 manager: ships no wazuh-syscheckd or logcollector command blocks. Nothing missing.
    • check_diff (rules 533/535, src/analysisd/dodiff.c) compares the stored output with strcmp() and needs no external command.
    • Indexer and dashboard (both versions): nothing missing. The 5.0 dashboard calls restorecon on its keystore file, finds none, and starts normally.

    Suggested follow-ups

    • Decide on netstat/last for 4.14: install net-tools and util-linux in the agent image, or drop the two <localfile> blocks from the wazuh-docker configuration. The packaged ossec.conf would still carry them.
    • Document which SCA checks do not apply to an agent running in a container.
    • Consider Requires: diffutils in packages/rpms/SPECS/wazuh-agent.spec, which today only declares Requires: coreutils. It covers any minimal host installing the RPM, not only these images.

    Verified against wazuh-docker 4.14.10 @ 76ddb88b and 5.0.0 @ ba299c39, wazuh/wazuh 4.14.10 @ 6e7835b4 and 5.0.0 @ 03bc3df6, and the images wazuh/wazuh-agent:4.14.10-rc1 (sha256:4d307cbb…) and wazuh-cicd/wazuh/wazuh-agent:5.0.0-latest (sha256:99c9c3c3…).

  3. linked a pull request that will close this issueAdd diffutils dependency - 4.14.11 #2786on Nov 2, 2026
  4. linked a pull request that will close this issueAdd diffutils dependency - 5.0.0 #2787on Nov 2, 2026
  5. moved this from In progress to Pending review in XDR+SIEM/Release 4.14.11on Nov 2, 2026
  6. mjbonacci commented on Nov 2, 2026

    @mjbonacci
    Member

    LGTM. Both PRs fix report_changes in a real deployment.

    • Add diffutils dependency - 4.14.11 #2786 (4.14.11) at 5c39ef6: built the agent and manager images from the PR branch with build-images.sh -v 4.14.10, since there are no 4.14.11 packages yet (that is also why build-docker-images fails on this PR and on every 4.14.x branch). fim_diff_changes.c is the same blob on 4.14.10 and 4.14.11. Against the same build of the base (8d86768), with report_changes on /fimtest in both the agent and the manager: the base gives (6714) and syscheck.diff: null on both, while the PR has no (6714) and the rule 550 alert carries the appended line on both.
    • Add diffutils dependency - 5.0.0 #2787 (5.0.0) at 5b6d5eb: single-node stack from the PR branch with the images its CI pushed (agent sha256:f815b8f7…), set up with certificates-conf.sh and credentials-conf.sh as the CI does, and two agents enrolled with tokens as in the agent guide: the PR's and the pre-fix 5.0.0-latest (sha256:99c9c3c3…). After the same change, the pre-fix agent gives (6714) and no file.diff, while the PR agent has file.diff in both wazuh-events-v5-system-activity and wazuh-findings-v5-system-activity. The 5.0 manager image has no wazuh-syscheckd, so it needs no change.

    Both branches are up to date with their base, and the CHANGELOG lines are in the right section.

    Evidence

    4.14 (arm64, wazuh-agent/wazuh-manager 4.14.10-1 from pre-release)

    base 8d86768   diff: NOT-FOUND, diffutils not installed (agent and manager)
      agent    2026/11/02 17:17:27 wazuh-syscheckd: ERROR: (6714): Command diff output an error
      manager  2026/11/02 17:17:27 wazuh-syscheckd: ERROR: (6714): Command diff output an error
      agent=agt-base41411   rule=550 event=modified size 11->39 diff=None
      agent=mgr-base41411   rule=550 event=modified size 11->39 diff=None
    
    PR 5c39ef6     diff: /usr/bin/diff, diffutils-3.8-1.amzn2023.0.2 (agent and manager)
      agent: no 6714 / manager: no 6714
      agent=agt-pr2786  rule=550 event=modified size 11->36 diff='1a2\n> marker-pr2786-1793639904\n'
      agent=mgr-pr2786  rule=550 event=modified size 11->36 diff='1a2\n> marker-pr2786-1793639904\n'
    

    5.0.0 (same stack, two agents, two changes each)

    prefix: 2026/11/02 17:57:34 wazuh-syscheckd: ERROR: (6714): Command diff output an error
    pr2787: no 6714
    events   | agent-prefix | diff=None
    findings | agent-prefix | Wazuh FIM - File modified - /fimtest/keep.txt | diff=None
    events   | agent-pr2787 | diff='1a2\n> marker-pr2787-1793642223\n'
    findings | agent-pr2787 | Wazuh FIM - File modified - /fimtest/keep.txt | diff='1a2\n> marker-pr2787-1793642223\n'
    events   | agent-pr2787 | diff='2a3\n> second-pr2787-1793642334\n'
    findings | agent-pr2787 | Wazuh FIM - File modified - /fimtest/keep.txt | diff='2a3\n> second-pr2787-1793642334\n'
    events   | agent-prefix | diff=None
    findings | agent-prefix | Wazuh FIM - File modified - /fimtest/keep.txt | diff=None
    
  7. moved this from Pending review to Pending final review in XDR+SIEM/Release 4.14.11on Nov 2, 2026
  8. moved this from Pending final review to Done in XDR+SIEM/Release 4.14.11on Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions