Skip to content

Fix: Disable in-pod placement for podman-compose by default#4381

Open
MarkTwoFive wants to merge 2 commits into
getsentry:masterfrom
MarkTwoFive:fix/podman-compose-deployment
Open

Fix: Disable in-pod placement for podman-compose by default#4381
MarkTwoFive wants to merge 2 commits into
getsentry:masterfrom
MarkTwoFive:fix/podman-compose-deployment

Conversation

@MarkTwoFive

Copy link
Copy Markdown

Hi Sentry community 👋🏼

When deploying self-hosted Sentry using rootless Podman and podman-compose instead of Docker/Docker compose, the containers may be placed inside a Pod.
This causes issues because the ad-hoc container runs performed by the install/upgrade script will fail if the dependencies are placed inside Pods.

Either:

  • The run command is rejected because the launched container is not in a Pod (see previously used work-around) and its dependencies are in a Pod.
  • The run command is rejected because the launched container is in a different (newly created) Pod than its dependencies from the stack.

The default behavior of podman-compose has changed from one stable version to another where containers are now placed inside Pods by default.
This PR aims to solve this compatibility issue by defining not being in a Pod as the default for this compose stack.

Background information

I'm currently deploying Self-Hosted Sentry on a RHEL VM with rootless podman and podman-compose being mandated by policy. I've gotten things running and I've even been able to make the deployment reproducible using Ansible.

Since Docker / Docker compose is not available to me, I am not able to verify whether Docker setups are un-touched by these changes. I am hoping somebody running Docker or the tests can verify that for me.

Left-out changes

Since Podman runs rootless (which means all containers run inside some user namespace) and the machine has SELinux enabled, there were additional changes needed to make things run. These involve mainly file permissions and se_labels.
Please let me know If I should even contribute this as it appears to be a rather niche deployment environment.

There's also one issue where the pip install https://(....) in a Dockerfile fails because this file does not take custom CA certificates into account. I might feel like contributing some fix for this too at a later time.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

Comment thread install/dc-detect-version.sh
@aldy505

aldy505 commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

I'll take a detailed look on your PR later, but a quick question:

There's also one issue where the pip install https://(....) in a Dockerfile fails because this file does not take custom CA certificates into account. I might feel like contributing some fix for this too at a later time.

You're saying, because you have a firewall that intercepts the TLS certificate? Or are you behind a corporate proxy?

@MarkTwoFive

Copy link
Copy Markdown
Author

You're saying, because you have a firewall that intercepts the TLS certificate? Or are you behind a corporate proxy?

We are behind a proxy and the pip command fails with certificate chain issues. For the time being, I've solved the issue by defining "github.com" as a trusted host. I am still working on a better solution and will create another PR when I have one.

@aldy505

aldy505 commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

You're saying, because you have a firewall that intercepts the TLS certificate? Or are you behind a corporate proxy?

We are behind a proxy and the pip command fails with certificate chain issues. For the time being, I've solved the issue by defining "github.com" as a trusted host. I am still working on a better solution and will create another PR when I have one.

How are you updating the self-hosted files? Is it via git checkout as seen on the docs? If you are, is it more beneficial for you if we use git submodule for the nodestore s3 plugin?

@MarkTwoFive

Copy link
Copy Markdown
Author

You're saying, because you have a firewall that intercepts the TLS certificate? Or are you behind a corporate proxy?

We are behind a proxy and the pip command fails with certificate chain issues. For the time being, I've solved the issue by defining "github.com" as a trusted host. I am still working on a better solution and will create another PR when I have one.

How are you updating the self-hosted files? Is it via git checkout as seen on the docs? If you are, is it more beneficial for you if we use git submodule for the nodestore s3 plugin?

To be specific, we're using Ansible git module. But yes, this would be a git checkout that is happening. I believe having a submodule or just checking out the dependency repository in one of the install steps would be working for us. Since the trust and proxy configuration would be the in the host terminal in that case.

I believe the current build just does not work because the Dockerfile doesn't include the custom CA image enhancements when it is built. To my knowledge, the nodestore is a required component, is that right? If it's not needed for the errors-only profile, just a toggle to skip its installation would be enough for us too.

@aldy505

aldy505 commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

You're saying, because you have a firewall that intercepts the TLS certificate? Or are you behind a corporate proxy?

We are behind a proxy and the pip command fails with certificate chain issues. For the time being, I've solved the issue by defining "github.com" as a trusted host. I am still working on a better solution and will create another PR when I have one.

How are you updating the self-hosted files? Is it via git checkout as seen on the docs? If you are, is it more beneficial for you if we use git submodule for the nodestore s3 plugin?

To be specific, we're using Ansible git module. But yes, this would be a git checkout that is happening. I believe having a submodule or just checking out the dependency repository in one of the install steps would be working for us. Since the trust and proxy configuration would be the in the host terminal in that case.

I believe the current build just does not work because the Dockerfile doesn't include the custom CA image enhancements when it is built. To my knowledge, the nodestore is a required component, is that right? If it's not needed for the errors-only profile, just a toggle to skip its installation would be enough for us too.

It's a required component for all profile. Hmm, should we convert this into a Git submodule to fight with this issue, then? This has been a problem since a while back, and people were trying to get their way around pip's TLS validation.

@MarkTwoFive

Copy link
Copy Markdown
Author

You're saying, because you have a firewall that intercepts the TLS certificate? Or are you behind a corporate proxy?

We are behind a proxy and the pip command fails with certificate chain issues. For the time being, I've solved the issue by defining "github.com" as a trusted host. I am still working on a better solution and will create another PR when I have one.

How are you updating the self-hosted files? Is it via git checkout as seen on the docs? If you are, is it more beneficial for you if we use git submodule for the nodestore s3 plugin?

To be specific, we're using Ansible git module. But yes, this would be a git checkout that is happening. I believe having a submodule or just checking out the dependency repository in one of the install steps would be working for us. Since the trust and proxy configuration would be the in the host terminal in that case.
I believe the current build just does not work because the Dockerfile doesn't include the custom CA image enhancements when it is built. To my knowledge, the nodestore is a required component, is that right? If it's not needed for the errors-only profile, just a toggle to skip its installation would be enough for us too.

It's a required component for all profile. Hmm, should we convert this into a Git submodule to fight with this issue, then? This has been a problem since a while back, and people were trying to get their way around pip's TLS validation.

I would say that would be one way. Another potential solution would be to allow image enhancement earlier in the Docker file which could be used to reconfigure pip so that it works with private/enterprise CAs.
Having a submodule requires less configuration but may confuse some customers who are not familiar with submodules.

- Some newer Podman versions always create Pods for the compose stacks containers to be placed in. This breaks the initialization commands as their containers are either placed in a new pod or already include the ``--in-pod=false`` option and are not in a Pod at all. Both variants will not be started by Podman since the containers dependencies would be located in a different pod.
- Pod-creation is now disabled by default and does not need to be specified in CLI anymore
@MarkTwoFive MarkTwoFive force-pushed the fix/podman-compose-deployment branch from cfee360 to 0c16353 Compare July 9, 2026 16:45
@MarkTwoFive

Copy link
Copy Markdown
Author

// Rebased onto master

# prints its pod id to stdout which is messing with the output that we
# rely on various places such as configuration generation
dcr="$dc --profile=feature-complete --in-pod=false run --rm"
dcr="$dc --profile=feature-complete run --rm"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Removing the --in-pod=false flag from the dcr alias risks corrupting script output, as the replacement global x-podman setting may not apply to podman-compose run commands.
Severity: HIGH

Suggested Fix

Restore the --in-pod=false command-line flag to the dcr variable definition in install/dc-detect-version.sh. This explicitly prevents podman-compose from printing pod IDs to stdout, ensuring that scripts capturing the command's output are not broken, rather than relying on an unverified global setting.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: install/dc-detect-version.sh#L54

Potential issue: The `--in-pod=false` flag was removed from the `dcr` command alias,
relying instead on a global `x-podman: in_pod: false` setting in the compose file. There
is uncertainty whether `podman-compose run` commands respect this global setting. If
they do not, `podman-compose` will create a pod and print its ID to standard output.
This will corrupt the output captured by various installation scripts, such as
`install/setup-js-sdk-assets.sh`, which expect clean JSON data. This would lead to
failures during the installation process.

Also affects:

  • docker-compose.yml:123~123

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants