Keeping Mutable Tags Like dev and latest Alive Under an ECR Lifecycle Policy

A Kubernetes Deployment in our development environment stopped starting with ImagePullBackOff. Looking at ECR, there was no image tagged dev.

$ aws ecr describe-images --repository-name app --image-ids imageTag=dev
An error occurred (ImageNotFoundException) ... The image with imageId {imageTag:'dev'} does not exist

ECR pull tokens expire after 12 hours, so authentication was the first suspect, but the Secret had just been refreshed and another container in the same Pod pulled fine. It was not auth — the image itself was gone. The lifecycle policy had deleted it.

Background: two kinds of tags on one image

Every build in this repository pushes two kinds of tags onto a single image.

dev and 1.12.370-dev from the same build point at the same digest. That is what bites.

Cause: ECR expires images, not tags

The policy at the time of the incident had only two rules.

{"rules":[
  {"rulePriority":1,"description":"Keep 3 images with version tags",
   "selection":{"tagStatus":"tagged","tagPatternList":["*.*"],"countType":"imageCountMoreThan","countNumber":3},
   "action":{"type":"expire"}},
  {"rulePriority":2,"description":"Delete untagged images",
   "selection":{"tagStatus":"untagged","countType":"sinceImagePushed","countUnit":"days","countNumber":1},
   "action":{"type":"expire"}}
]}

*.* means "tags containing a dot" and is aimed at version tags. dev has no dot, so it does not match. At a glance dev looks safe.

But what ECR expires is an image (a digest), not a tag. 1.12.370-dev does match *.*. After three production builds, 1.12.370-dev falls outside the "newest three", that image gets expired, and the dev tag that was sitting on the same image disappears with it.

The less often the development environment is built compared to production, the more likely this is. In our case, four production builds ran within a month of the last development build.

This behaviour is documented in the lifecycle policy evaluation rules: an image is expired by exactly one or zero rules, and an image that matched the tag conditions of one rule cannot be expired by a rule of lower priority. That second half is what the fix rests on.

Fix: protect mutable tags with a high-priority keep-1 rule

For each mutable tag, add a "keep only the newest one" rule at a higher priority than the version-tag rule.

{"rules":[
  {"rulePriority":1,"description":"Keep the newest latest",
   "selection":{"tagStatus":"tagged","tagPatternList":["latest"],"countType":"imageCountMoreThan","countNumber":1},
   "action":{"type":"expire"}},
  {"rulePriority":2,"description":"Keep the newest dev",
   "selection":{"tagStatus":"tagged","tagPatternList":["dev"],"countType":"imageCountMoreThan","countNumber":1},
   "action":{"type":"expire"}},
  {"rulePriority":3,"description":"Keep 3 generations of version tags",
   "selection":{"tagStatus":"tagged","tagPatternList":["*.*"],"countType":"imageCountMoreThan","countNumber":3},
   "action":{"type":"expire"}},
  {"rulePriority":4,"description":"Delete untagged after 1 day",
   "selection":{"tagStatus":"untagged","countType":"sinceImagePushed","countUnit":"days","countNumber":1},
   "action":{"type":"expire"}}
]}

imageCountMoreThan: 1 expires everything beyond the first, so the newest image carrying the mutable tag survives. Once the mutable tag moves to a newer image, the old image is left with only its version tag and falls back under rule 3's generation management; when it drops out of there too it becomes untagged and rule 4 removes it. The baton passes cleanly between rules.

When you add an environment and therefore a new mutable tag, add its keep-1 rule as well. Forget it, and that environment eventually hits the same failure.

Dry-run with preview before applying

put-lifecycle-policy takes effect the moment it is applied, and matching images are gone within 24 hours. Put a wrong policy and there is no time to notice. Feed the same JSON to start-lifecycle-policy-preview first and read the result.

policy=$(cat policy.json)
repo=app

aws ecr start-lifecycle-policy-preview \
  --repository-name "$repo" --lifecycle-policy-text "$policy"

# The preview is asynchronous. Wait for COMPLETE instead of a fixed sleep
until [ "$(aws ecr get-lifecycle-policy-preview --repository-name "$repo" \
           --query status --output text)" = COMPLETE ]; do sleep 5; done

# previewResults lists only the images that would be expired
aws ecr get-lifecycle-policy-preview --repository-name "$repo" \
  --query 'previewResults[*].{tags:imageTags,rule:appliedRulePriority}' --output json

aws ecr put-lifecycle-policy \
  --repository-name "$repo" --lifecycle-policy-text "$policy"

If previewResults contains an image tagged latest or dev, the priority or the pattern is wrong.

Setting this up by hand in the console leaves nothing reproducible, and the next repository you create repeats the same accident. Wrap the preview-then-put sequence in a shell script with the policy JSON in a heredoc and keep it in the repository; changing a rule then means editing and re-running it.

Something the preview revealed: protected images still count

The preview gave a result I did not expect. With four images carrying version tags — 1.12.359, 1.12.365, 1.12.368 (latest) and 1.12.370-dev (dev) — the one listed for expiry was 1.12.359.

If the dev image were excluded from rule 3's evaluation, rule 3 would see three version-tagged images and delete nothing. What actually happens is that the dev image is counted when picking the "newest three", is then protected and survives, and one older image, 1.12.359, is pushed out instead.

So "an image matched by a higher-priority rule is not expired by a lower-priority rule" is true, but it does not mean "excluded from the lower-priority rule's count". While the development image is newer than production, you keep one fewer production generation than the number you configured. It does not weaken the protection of the mutable tags, so I left it as is; if you need the generation count exactly, raise countNumber by the number of mutable tags.

The option I did not take: no version tag on dev

If development images only got dev and never 1.12.N-dev, nothing would match *.* and the old policy would already be safe. Old images left behind by the moving tag become untagged and vanish in a day. It is a workflow change only, with no policy edit.

The cost is losing version-tag based tracking. Other repositories already protect their mutable tags with keep-1 rules, so I went with the policy fix for consistency.

Deleted images do not come back

Fixing the policy does not resurrect the deleted dev. The development image has to be rebuilt and pushed.

Do it in that order: restore the environment with a build and push, then fix the policy. Fixing the policy first brings nothing back, and rebuilding while leaving the policy alone just means the image disappears again after three production builds.