Podman push command is failing with "Digest did not match" error

[tarun360]

I had a image stored in a NFS partition. I had a script which used to pull this image to containers-storage and push it to a remote registry, This used to work fine until today when I started getting the below error during push operation.

Error: writing blob: Patch "{some_patch}": happened during read: Digest did not match, expected sha256:{some_image_digest}, got sha256:{some_other_image_digest}

I was able to get around this error by removing the previous occurrence of this image in containers-storage (using podman rmi localhost/image-name), and then running the script again.

Possible explanation I could think of is: Podman skipped pulling layers already present in containers-storage. The image layers present in containers-storage got corrupted somehow. While trying to push the image to remote registry podman re-calculated digests of image layers and they didn't match the digest present in manifest, which gave the above error. Removing the image from containers-storage got rid of corrupted image layers which solved the problem.

Does this reasoning sound plausible? and how could one deal with such cases in future.

[vrothberg]

Thanks for reaching out, @tarun360.

It's hard to point the finger without knowing more about what the script does. @mtrmac have you seen such a symptom before?

[tarun360]

The script essentially did

# Pull image from nfs folder
podman pull oci-archive:/path/to/nfs/folder
# Push image to remote registry
podman push localhost/image-name docker://some-remote-registry/image-name

I get the above digest mismatch error during push operation. I am on a rhel8 node.

I saw this link, which refers to a known image push issue with NFS volume on rhel platforms. I am not sure if this is related.

[mtrmac]

That’s either c/storage, or the backing filesystem corrupt — or, possibly, some bug in storing and/or reconstructing layers in c/storage.

The way to diagnose this would be to

• get the corresponding known-good layer, (uncompress it if relevant) to obtain a tar file
• find the relevant c/storage layer, extract it without involving Podman (probably by compiling https://github.com/containers/storage/blob/main/cmd/containers-storage and finding the right command to obtain the layer contents.), to obtain another tar file
• Compare the two (and the reported digests) to see if/how they differ
• The difference can then either be attributed to unexpected on-filesystem contents (and that would need to be diagnosed), or to a correct filesystem somehow incorrectly processed (and that would need to be diagnosed). How exactly that would be diagnosed depends on the corruption in question, and might not necessarily be easy.

All of that requires access to the previously-corrupt c/storage store.

[lddlww]

@mtrmac does chmod -R a+rx affect digest?

[mtrmac]

Impossible to tell. chmod a+rx of what?

[lddlww]

for public images dir

Public image directories face permission issues because Podman defaults to setting file permissions to 600 when running containers. This causes rootless users to be unable to access public images. To resolve this, we must execute chmod a+rx on the public image directory. However, this introduces a new problem: rootless users can no longer push images to the repository due to digest did not match caused by the altered permissions.

but root user works fine even chmod public image dir
am i missing something or how can i do to solve it?

thanks in advantage

[mtrmac]

Please file a specific report with detailed steps to reproduce. I suspect force_mask might help, but there might be more to it.

[mtrmac]

#26690 , for reference.