Digest immutability issue with `quay.io/nginx/nginx-unprivileged` image

[aydosman]

Describe the bug

When using the NGINX Unprivileged Docker image from quay.io, it appears that during releases the digests of all tags are updated. As a result even when attempting to use a specific pinned version (e.g., quay.io/nginx/nginx-unprivileged:1.27.3-alpine@sha256:3092a71e4222a73893547dad52bbcf259582a6dd40c8b616e5bf44bcca3f01ff), the digest changes, making it impossible to pull the original image.

Expected behavior
The digest for a pinned version should remain immutable and unchanged, allowing users to reliably pull specific versions of the image regardless of subsequent releases.

Additional context
We observed that other related images such as the following, behave as expected with tags and digests remaining consistent on release:

nginx-prometheus-exporter
nginx-ingress
nginx-ingress-operator

Is there a reason why the digests for the nginx-unprivileged image change across releases? Or is this an oversight? This behaviour significantly impacts our environments.

[aydosman]

@alessfg -Any updates on this? I am asking you directly because I see that you are running the GitHub Action.

[mahesh-hegde]

I am observing this as well. Furthermore, I cannot search the old digest in the "untagged images" section of the ghcr because of this workflow. So this prevents me from even referring to this digest at all (like ghcr.io/nginxinc/nginx-unprivileged:1.27.3-bookworm@sha256:3092a71e4222a73893547dad52bbcf259582a6dd40c8b616e5bf44bcca3f01ff).

So it's not ideal for reproducibility.

I think it's inevitable the SHA would change because docker cache would not always be available and some OS dependent details (like the OS packages themselves, or anything that is randomly generated) may upgrade.

However is it required that you need to release image for a given nginx version eg: 1.27.3 multiple times and overwrite the existing package?

Or can we keep older digests?

Thanks

[alessfg]

I really don't have a good solution for this issue right now. Images are rebuilt on a weekly basis to avoid potential security issues and keep the base image updated.
If I update the image, the digest is also going to change no matter what. I will note that this is also the case with our core NGINX image and probably almost every other image out there, but they just happen at a much lower rate than what you see in this image.

Re the second issue mentioned here -- once upon a time I didn't cleanup untagged images, but that lead to issues with the image repositories getting full. I guess I could explore only deleting untagged images that are "x" time old, but I don't really have a timeline for when I could start looking into it. PRs are always welcome if any of you have any ideas!

[stevehipwell]

@alessfg it's a best practice to reference container images by their digest even if the tags are immutable, for mutable tags using digests is essential. This is primarily a security concern around the supply chain, but it's also required to have a deterministic system and matching pods across nodes.

Given the current pattern you're using if you add an additional <major>.<minor>.<patch>-<n> tag when building where <n> is incremented from the last build (next_n="$(crane ls --omit-digest-tags ghcr.io/nginxinc/nginx-unprivileged | grep -E '<major>\.<minor>\.<patch>-\d+' | jq -sRr '[split("\n") | .[] | select(. != "")] | sort | .[-1] | match("[0-9]+$").string | tonumber | . += 1')"). Then when you move the <major>.<minor>.<patch> tag the digest will no longer be cleaned up as it will still be tagged.

[alessfg]

Right, but this doesn't remove the issue of some registries being hard capped to only a set amount of images before running into issues. Not only that, but the goal is to keep these images and the tags identical to the core Docker NGINX images, and changing the tag pattern would break that.

Changing the cleanup algorithm to be time based instead of automatically deleting any untagged images is probably the way to go, but I need to find some time to dig into it.

[stevehipwell]

Right, but this doesn't remove the issue of some registries being hard capped to only a set amount of images before running into issues

@alessfg no it doesn't but it's not attempting to do so; it does fix the issue of this image not being consumable by industry best practices. Dealing with cleanup is a separate concern, but one that's easy enough to do as a GitHub Actions CRON job; for example a nightly check using crane to list the tags and if they're close to the limit (which is what, and are we talking tags or layers or total data?) clean up some of them based on a rule (e.g. remove a batch of the oldest <major>.<minor>.<patch>-<n> tags).

Not only that, but the goal is to keep these images and the tags identical to the core Docker NGINX images, and changing the tag pattern would break that.

The NGINX Docker library images aren't removed once published, surely that's the most important compatibility feature? That said the pattern I proposed above isn't mutually exclusive to your desire as it's not modifying your current tag pattern but is purely additive.

[alessfg]

Docker hub images for this image should not be being removed by any cleanup scripts. If they are, it's an issue with Docker hub itself, not this repo.

I won't go into details re the AWS ECR/GHCR restrictions beyond saying that they are quite strict. I would rather not have to deal with image cleanup at all.

My current plan still is to modify the cleanup scripts to delete any untagged images above two years old. I have no ETA for when it will be done though. This repo is very much maintained on a best effort basis, but rest assured this is the next "major" update I'll be working on for this project.

[alessfg]

I've gone ahead and opened a PR that should hopefully address this issue (#297). I will however note that I do not delete Quay images at all since Quay does its own garbage collection. The main difference between the other projects originally mentioned and this one is that the other projects tend to only create a new image whenever a new release happens, whilst this image gets rebuilt on a weekly basis to account for any potential CVEs.

Just in case, for now I would suggest pulling images from AWS ECR, Docker hub, or GHCR using the digest since this PR should ensure that each digest lives for at least 2 years (which is time enough to migrate to a new release/digest before the old one gets permanently removed).