Skip to content

[Bug]: Stale factset — facts never update in PuppetDB for agent self-compiling on the same host as Puppet Server #520

Description

@nrondel

Is this a critical security issue?

  • This is not a security issue.

Describe the Bug

Facts submitted by an agent running on the same host as the compiling Puppet Server (self-compile / loopback-style catalog request) never update in PuppetDB after the first factset, despite successful replace_facts commands logged on every subsequent run — reproduced even after changing the agent's certname to differ from the server's own certname, ruling out certname collision as the cause

Expected Behavior

On every puppet agent -t run, freshly resolved facts (both core facts like system_uptime/load_averages and external/custom facts) should be persisted to PuppetDB, replacing the previous factset — regardless of whether the compiling Puppet Server and the requesting agent are the same host.

Steps to Reproduce

  1. Set up a Puppet Server node that also runs as its own agent (server = itself, e.g. via 127.0.0.1 or its own hostname/IP).
  2. Run puppet agent -t on that node multiple times, a minute or more apart.
  3. Query PuppetDB for a volatile fact on that node, e.g.:
    curl -s "http://localhost:8080/pdb/query/v4/nodes//facts/system_uptime"
  4. Observe that the returned value never changes between runs, even though the actual system uptime has advanced and each run logs a successful catalog application and a 'replace facts' command with no errors in puppetserver.log.
  5. The factset only refreshes correctly after the row is deleted from PostgreSQL's factsets table (DELETE FROM factsets WHERE certname = '<certname>') and a fresh puppet agent -t is run — i.e. INSERT works, UPDATE does not.
  6. Reproduced twice independently on the same host: once with an external custom fact (client), once with another (puppet_agent_disabled), and confirmed again after fully re-registering the agent under a different certname than the server's own certname (ruling out certname collision as the root cause).

Environment

  • openvox-server / openvoxdb version: 8.13.0 → also reproduced after upgrading to 8.14.1
  • OS: Ubuntu 24.04
  • PostgreSQL 16
  • Topology: single-node Puppet Server acting as its own Puppet agent (typical for small/all-in-one OpenVox GUI deployments)

Additional Context

Inspected the factsets table directly in PostgreSQL. The stable/volatile JSONB columns and the overall hash never change across affected runs, while timestamp/producer_timestamp DO update correctly — meaning PuppetDB acknowledges and logs each submission, but doesn't persist the new content for this specific certname/row. Other nodes in the same infrastructure (regular agents, not self-compiling) update their facts normally on every run.

This may be specific to the self-compile / loopback catalog-request code path in Puppet Server, possibly involving stale in-request fact caching that bypasses the normal PuppetDB write path for the compiling node's own facts.

Relevant log output

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions