Skip to content

Unable to build some images from Docker - While performing build: conveyor failed to get: open #4088

Closed
Closed
Unable to build some images from Docker - While performing build: conveyor failed to get: open#4088
Labels
dependenciesPull requests that update a dependency fileneeds investigation

Description

@asiomchen
asiomchen
opened on Apr 10, 2026

Before you report an issue...

Version of Singularity

What version of Singularity are you using? Run:

$ singularity --version
singularity-ce version 4.4.1

Describe the bug

Trying to build some images from Docker images fails e.g pixi and pixi-derived images

To Reproduce

Steps to reproduce the behavior:

docker pull ghcr.io/prefix-dev/pixi:latest
singularity build pixi.sif docker-daemon:ghcr.io/prefix-dev/pixi:latest

Current output:

INFO:    Starting build...
INFO:    Fetching OCI image...
FATAL:   While performing build: conveyor failed to get: open /home/anton/.singularity/cache/blob/blobs/sha256/7574239a4ba2e6a23ea4ff665e3a4ca9f01f8f2dafb4ff6b1e4808e06df0c8e5: no such file or directory

Expected behavior

Build finishes succesfully

OS / Linux Distribution

Which Linux distribution are you using? Is it up-to-date?
It is WSL2 Ubuntu

PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Installation Method

Compiled from source, the error also exists if installed from DEB

Activity

  1. asiomchen
    added
    bugSomething isn't working
    on Apr 10, 2026

  2. dtrudg commented on Apr 13, 2026

    @dtrudg
    dtrudg
    on Apr 13, 2026
    Contributor

    There is an incompatibility between the google/go-containerregistry library SingularityCE uses to access the docker-daemon, and newer versions of docker.

    Which version of docker are you using (docker version)? As this would allow us to confirm the issue.

    As a workaround use docker save to export the image, and then run the singularity build from using docker-archive:<filename>

  3. dtrudg
    added
    dependenciesPull requests that update a dependency file
    on Apr 13, 2026

  4. asiomchen commented on Apr 13, 2026

    @asiomchen
    asiomchen
    on Apr 13, 2026
    Author

    Thank you for the prompt reply.

    Docker version 28.5.1, build e180ab8

    Confirmed, the code below works:

    docker save ghcr.io/prefix-dev/pixi:focal -o pixi-image.docker
singularity build pixi.sif docker-archive:pixi-image.docker

    Output:

    INFO:    Starting build...
INFO:    Fetching OCI image...
INFO:    Extracting OCI image...
INFO:    Inserting Singularity configuration...
INFO:    Creating SIF file...
INFO:    Build complete: pixi.sif

    @dtrudg can you share some more context, what is the affected docker version range? In there some issues on google/go-containerregistry I can track to learn about the fix process?

  5. dtrudg commented on Apr 13, 2026

    @dtrudg
    dtrudg
    on Apr 13, 2026
    Contributor

    Glad that the workaround worked.

    @dtrudg can you share some more context, what is the affected docker version range? In there some issues on google/go-containerregistry I can track to learn about the fix process?

    It is a repeat issue, which occurs for different docker versions as the docker daemon / client API changes relatively frequently. See e.g.

    Sometimes the streaming output from the daemon, which the ggcr library consumes, also changes with these API changes.

    Unfortunately the situation around docker v28 / v29 is especially confusing, as it seems some breaking changes were implemented and then reverted (I'm not sure exactly when).

    General advice is to just build directly from docker:// sources with singularity or do the docker save singularity build docker-archive: dance - as the daemon source is fragile, outside of our control upstream. We do not test against multiple docker versions routinely - only the version in the current CircleCI ubuntu machine image is routinely tested as a docker-daemon source.

  6. dtrudg commented on Apr 13, 2026

    @dtrudg
    dtrudg
    on Apr 13, 2026
    Contributor

    FWIW - everything is working on a local machine, for me, with docker v29.4.0 - which is the latest. So it appears that with the ggcr update to use the v29 docker client, use against docker versions <29 may now be having issues. This isn't something we can address I'm afraid.

    I'll look into getting the version compatibility issues mentioned in documentation / singularity help text etc.

    02:47 pm $ docker pull ghcr.io/prefix-dev/pixi:latest
latest: Pulling from prefix-dev/pixi
689b91d88a0f: Pull complete 
1420face9adb: Pull complete 
18c4f161c388: Pull complete 
72b65e0f9413: Pull complete 
Digest: sha256:d2ba1b9be8edc147d2ef5fbf25db78718e0a80daef3b9417b36e05cb16452a52
Status: Downloaded newer image for ghcr.io/prefix-dev/pixi:latest
ghcr.io/prefix-dev/pixi:latest

02:48 pm $ singularity build pixi.sif docker-daemon:ghcr.io/prefix-dev/pixi:latest
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Fetching OCI image...
INFO:    Extracting OCI image...
INFO:    Inserting Singularity configuration...
INFO:    Creating SIF file...
INFO:    Build complete: pixi.sif

02:48 pm $ docker version
Client: Docker Engine - Community
 Version:           29.4.0
 API version:       1.54
 Go version:        go1.26.1
 Git commit:        9d7ad9f
 Built:             Tue Apr  7 08:35:57 2026
 OS/Arch:           linux/amd64
 Context:           default
  7. dtrudg
    added
    documentationImprovements or additions to documentation
    wontfixThis will not be worked on
    and removed
    bugSomething isn't working
    on Apr 13, 2026

  8. dtrudg commented on Apr 16, 2026

    @dtrudg
    dtrudg
    on Apr 16, 2026
    Contributor

    Further digging suggests it might be related to containerd being used for snapshots... google/go-containerregistry#1954 (closed stale, but seems likely this is the real issue).

    Reason then for different behaviour on my system would be docker has been updated over time... so didn't switch to the containerd snapshotter (see https://docs.docker.com/engine/storage/containerd/)

    @asiomchen - does your docker version output confirm it's using containerd snapshotter? e.g.

    docker info -f '{{ .DriverStatus }}'
[[driver-type io.containerd.snapshotter.v1]]

    If confirmed.. this is still an upstream issue we can't really work around, short of scanning and manually fixing the layout dir we use to cache blobs. This would be an unreasonably lengthy / IO intensive step on any system that has pulled more than a couple of images.

  9. dtrudg
    added
    needs investigation
    and removed
    documentationImprovements or additions to documentation
    wontfixThis will not be worked on
    on Apr 16, 2026

  10. asiomchen commented on Apr 16, 2026

    @asiomchen
    asiomchen
    on Apr 16, 2026
    Author

    Yes, this is exactly the case for me

    ➜  ~ docker --version
Docker version 29.4.0, build 9d7ad9f
➜  ~ docker info -f '{{ .DriverStatus }}'
[[driver-type io.containerd.snapshotter.v1]]

    Even after docker update and singularity/docker cache cleanup the error still exists in my system. (at least workaround still works)

  11. dtrudg commented on Apr 16, 2026

    @dtrudg
    dtrudg
    on Apr 16, 2026
    Contributor

    Thanks. That does appear to be the underlying issue then. We can't really tackle it short of fixing the dependency upstream or forking it. I'll try to propose a PR upstream at some point.

  12. dtrudg
    added 2 commits that reference this issue on May 27, 2026

    e2e: skip DOCKER/docker_host tests if containerd snapshotter

    182364e

    e2e: skip DOCKER/docker_host tests if containerd snapshotter

    9be036e
  13. dtrudg
    closed this as completedon Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dependenciesPull requests that update a dependency fileneeds investigation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions