Skip to content

[Bug]: container image push fails to register manifest digests referenced in OCI Index, causing 404s on pull #1001

@venim

Description

@venim

I have done the following

  • I have searched the existing issues
  • If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

Reproduction Steps:

  1. Build for a specific arch: container build --arch amd64 -t my-reg/my-image:v1
  2. Push to registry: container image push my-reg/my-image:v1
  3. Attempt to pull from another machine: crictl pull my-reg/my-image:v1

Observed Registry Logs (Zot):

{"method":"GET","path":"/v2/my-image/manifests/sha256:<index-digest>","statusCode":200}
{"method":"GET","path":"/v2/my-image/manifests/sha256:<manifest-digest>","statusCode":404}

Observed K8s Events:

  • Failed to pull image "...": rpc error: code = NotFound desc = failed to pull and unpack image "...":
  • failed to copy: httpReadSeeker: failed open: content at .../manifests/sha256: not found

Workaround:

Exporting the image to a tarball and using skopeo copy to the registry fixes the issue, as skopeo correctly handles the manifest registration:

container image save my-reg/my-image:v1 -o image.tar
skopeo copy oci-archive:image.tar docker://my-reg/my-image:v1

Current behavior

When building an image for a specific architecture (e.g., --arch amd64) and pushing it to a registry, the container CLI creates and pushes an OCI Image Index (manifest
list). While the index itself is pushed successfully, the platform-specific manifest it references is not being correctly "registered" in the registry's manifest store.

When a client (like containerd) attempts to pull the image:

  1. It successfully fetches the OCI Index by tag.
  2. It identifies the correct manifest digest for the target architecture from the index.
  3. It attempts to fetch that specific manifest by digest: GET /v2//manifests/sha256:.
  4. The registry returns 404 Not Found.

Investigation shows the manifest content was uploaded as a blob, but the registry does not recognize it as a valid manifest endpoint, suggesting the push command is
missing the final PUT to the manifests functional endpoint for the sub-manifests.

Expected behavior

The push command should ensure all manifests referenced by the OCI Index are correctly registered at the registry's /manifests/ endpoint so they can be pulled by
content-addressable digest.

Environment

- OS: macOS Version 15.7.3 (24G419)
- Container: container CLI version 0.7.1 (build: release, commit: 420be74)

Relevant log output

N/A

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions