Repository navigation
[Discussion] Non-root interferes with helm chart #181
Description
Since #141, OpenVoxProject/openvox-helm-chart is facing the following issue:
OpenVoxProject/openvox-helm-chart#44
/container-entrypoint.d/20-use-templates-initially.sh: line 12: cd: /opt/puppetlabs/server/data/puppetserver/.puppetlabs/etc/puppet: No such file or directory
I'm reporting this here because the causes (and most fixes) also land here.
Long story short, /opt/puppetlabs/server/data/puppetserver is puppetserver's master-var-dir (separate from vardir (that tripped me up)). It also holds the non-root homedir. It is also a directory people commonly mount a volume over. By default the helm chart does this via the "serverdata" PVC. Prior to #141 this was fine because the container was rootful and the home data lived elsewhere. Now, we get the above error.
If you work around this in any way puppetserver still dies with another, different error:
Caused by: org.jruby.exceptions.LoadError: (LoadError) cannot load such file -- puppet
This is because in #141 the Ubuntu container changed to match the Alpine behavior where we use puppetserver gem install openvox instead of installing puppet via the .deb file. Meaning, we're installing the gems during build time rather than runtime, and it's saving to that same master-var-dir, so it's similarly getting overwritten by the PVC.
On top of all that, anyone's workflow that was relying on the previous homedir location (say, for example, you mounted files into .ssh/ or .gitconfig/) has broken. The same goes for anyone mounting gems to the old location instead of using puppetserver gem install. And even then the latter gets wiped on container reboot.
I'm opening this to discussion because there are multiple approaches, and none of them are perfect.
- Move the homedir and gemdir (maybe
puppetserver/appsinstead ofpuppetserver/data?)- The homedir is already set via
ENV HOMEto work around random UIDs, and thegem-pathandgem-homeare already set viapuppetserver.conf, so this would be easy enough. - If someone is relying on the old non-root paths they'll have to update it to the new non-root path.
- The homedir is already set via
- Specify
--confdir/--configfor all the puppet/puppetserver commands in entrypoint scripts- Less moving parts and easier to understand, imo
- Anything calling
puppetad-hoc inside the container might not return correct results (don't know why someone would do this) - We'd still have to move the gemdir
- Set data persistence on the serverdata PVC to default false
- Don't have to move the gemdir
- Another step in the long march towards immutability (of which I have strong feelings about)
- May break certain workflows. For example, if you're extracting the reports manually instead of sending it to PDB.
- Create an environment variable upstream for the confdir et al
- I'm seeing prior art for
run_dir - Blast radius is far larger if anything goes wrong
- We'd still have to move the gemdir
- Probably quite useful even outside of this use case
- Likely duplicating work in the XDG basedir PR
- I'm seeing prior art for
Let me know what people think.
Activity
rwaffen commentedon Jul 29, 2026
#153 is handling some of the points from here already, doesn't it? maybe we should rebase and use this?
for any other topic i'm not good enough. and dont want to paste just wall-of-text from some llm 😅
EDIT:
but if you think this could be somewhat benficial i can post what my codex thinks about this.
dotconfig404 commentedon Jul 29, 2026
Good analysis, thanks.
I am quite interested in hearing about the "march towards immutability" - that is basically the idea that one should simply not have a need to mount that dir in the first place?
Back when I wrote the PR to make this image OpenShift compatible, I also considered pinning
--confdirnamely to/etc/puppetlabs/puppet, but decided against it, as I was actually running into the issue you mentioned, as I was testing module-related things in the server usingpuppet module install. Besides, non-root users have the confdir in~/.puppetlabs/etc/puppetby convention and the $HOME for puppet is (at least in Debian and Red Hat systems) in/opt/puppetlabs/server/data/puppetserver, so/opt/puppetlabs/server/data/puppetserver/.puppetlabs/etcwould be the correct place for the confdir.So regarding the confdir location / homedir I would rather suggest an entrypoint script to recreate the structure at runtime and fail with a clear message when $HOME isn't writable.
Regarding the gem-home though, I would say relocation out of the master-var-dir makes sense, with subsequent addition to
puppetserver.confin the gem-path.Generally though, the best fix imo is upstream. This is something I also considered back then. IIRC the way to determine the confdir and other paths was pretty finicky, in both the ruby openvox/puppet gem and the clojure server code. I am happy to dig deeper into this and get back to speed, been some time since I investigated this. But yea, not sure how deep the rabbit hole goes here right now, a lot of historical baggage, e.g. the
cadirmoving and being inconsistent between server and gem was one thing that tripped me back then, besides all the obviouspuppetbranding which I wish we could move away from towardsopenvox.MirandaStreeter commentedon Jul 29, 2026
Ahhh I can't believe I missed #153, that is far more elegant. I kept trying to move the static content out of the vardir (and kept coming up with different approaches because none of them were very satisfying), but that PR just moves the vardir down a level. Much nicer. I'm closing this one in favor of that.
dotconfig404 commentedon Jul 29, 2026
I'd still be interested to hear about your "immutability march". Using the new directory means a migration, as the new var dir lives inside the currently mounted path. Would need some coordination with a PR in the helm repo as well to change the mount path to the new var dir. Certainly doable, but not convinced about a quick decision here.
Also, /opt/puppetlabs/server/data/puppetserver is still something that will easily get mounted anyways in an attempt to mount the vardir, even if sufficiently documented in the image doc. Moving the gem dir instead of the var dir and using a self-healing entrypoint script that fails with a clear error message on permission issues appears to me to be more frictionless.
MirandaStreeter commentedon Aug 5, 2026
I'm reopening this for further discussion.
I've had my fair share of customer interactions regarding large fleets of compiler servers. A running theme was wishing for puppetserver orchestration that could be scaled up and torn down as needed. And part of making that viable is the cliche "cattle, not pets" adage. The more we encourage treating container images like mini-VMs to be exec'd into and babied, the more we have to deal with statefulness and brittleness.
If I could wave a magic wand I'd make it possible to set the JAR as the entrypoint itself, load it up with environment variables, separate the puppet code into a sidecar, and rip out the CA components into its own container. From my perspective wrangling a dozen mountpoints for config is antithetical to the advantages containers bring.
Yes, so, it is mentioned in #153 that it's a breaking change. But from what I can tell it's just holding a
restartcounterand the facts (which the agent rewrites on its next run anyway). Maybe if you setreports = storeyou'll have something worth preserving, but doing that within a container rather than sending it out to the DB is treating containers like mini-VMs which, again, I feel strongly we should discourage at every opportunity.But even then the "Follow-up" in #153 alludes to the solution. I believe the helm chart can be updated for both the data relocation and image update simultaneously. When you update the
mountPathit shouldn't orphan the data. The storage exists in the volume independently of the pod. :)In either case I'd much prefer to avoid any kind of self-healing scripts. The more moving parts we create the more structural weak points we have to live with.
anthonysomerset commentedon Aug 11, 2026
Does anybody even do any of these workflows today that would be impacted by this change, because the way i read it this is probably the most elegant and yes anything that reduces the need for persistant volumes and/or increases immutability ftw in my humblest of opinions
dotconfig404 commentedon Aug 11, 2026
Agreed, and to take it further, I would really like to know if the "JAR-container" vision is something more feasible than it is assumed - what do we really need to mount, the SSL dir and the puppet code/data?
Sth like auth.conf is a bit tricky perhaps to set via ENV variables, but surely there's a solution there as well. Right now the image is pretty messy in any case with all the scripts and mountpoints, and I agree that another entrypoint script (aka the self-healing script I mentioned) would add to that.
anthonysomerset commentedon Aug 11, 2026
Isn't configmap the direct analog for complex configs that can't easily be broken into ENV variables anyway? The challenge there becomes the judgment call between complex enough to use configmap vs ENV vars
MirandaStreeter commentedon Aug 11, 2026
Oh, yes, configmaps are also perfectly cromulent. My story was more about taking steps to avoid the tech sin of creating a PVC just to mount a
*.conffile.I don't see us being able to whittle down the entrypoint scripts and gunning for the JRE itself until we start separating out components like the CA subsystem, but that's outside the scope of this issue and a discussion for 10.x and beyond.
Quite possibly? I honestly don't know. That's part of why I opened this discussion, to see what assumptions we can break. I'm perfectly happy with submitting that in place of the gem/home path shuffling.
dotconfig404 commentedon Aug 12, 2026
Yea, agreed. Mounting *.conf files is the real issue.
We are already gunning the JRE, actually. Right now it is the foreground script that does it. The point is more that the container is set up in a way that suggests it is not transient, a mini-VM: uses a root-drop layer (which is never used), a trap-and-pkill relay for SIGTERM instead of simply exec'ing the main process, and has a redundant (since OpenShift compatibility) service account and directory layout imitating a package install on a full OS.
So the point is not necessarily gunning the JRE directly, it is cutting out complexity.
dotconfig404 commentedon Aug 12, 2026
The CA subsystem could stay as it is btw, one could still exec in the container and do a puppetserver ca clean for instance if desired. Although it creates the mini-VM workflow again, and besides, iirc the ca tool uses the API anyways, so might as well use the tool locally instead.
Back on topic: turning off the PVC or getting rid of it entirely is the cleanest, yea. Could instruct users to use http or log instead of store then, if they dont want to use puppetdb. Or, create a PVC to the reports dir directy, instead of the parent dir.