Push to ECR fails intermittently with "failed to solve: failed to push * unexpected status from HEAD request to *: 403 Forbidden"

[i-am-gg]

I did my research before posting this issue again, and found out #983, now having gone through the discussion the solutions suggested

• It was a one off AWS problem
• Missing ecr:BatchGetImage permission to the user.

But, what I am facing is, it works, fails in the next build, works again on re-trigger, and then fails continuously for the next couple of build.

I am attaching my composite action file as well

name: "Docker Build & Push"
description: "Builds the Docker image and pushes it to ECR"
inputs:
    environment:
        description: "Environment"
        required: true
    ecr_account_id:
        description: "ECR Account ID"
        required: true
    ecr_region:
        description: "ECR Region"
        required: true
    service_name:
        description: "Service Name"
        required: true
    service_version:
        description: "Service Version"
        required: true
    docker_file:
        description: "Dockerfile to use"
        default: "Dockerfile"
        required: false
    context:
        description: "Docker build context directory"
        default: "."
        required: false
    secrets:
        description: "Secrets"
        required: true

runs:
    using: "composite"
    steps:
        - name: parse secrets
          shell: bash
          run: |
              echo "${{ inputs.secrets }}" > secrets.yaml
              keys=$(yq eval 'keys | .[]' secrets.yaml)
              for key in $keys; do
                value=$(yq eval ".$key" secrets.yaml)
                echo "$key=$value" >> $GITHUB_ENV
              done

        - name: Configure AWS credentials
          uses: aws-actions/configure-aws-credentials@v4
          with:
              aws-region: ${{ inputs.ecr_region }}
              aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }}
              aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }}

        - name: Log in to Amazon ECR
          id: login-ecr
          uses: aws-actions/amazon-ecr-login@v2

        - name: Form ECR URI
          shell: bash
          run: echo "ECR_URI=${{ inputs.ecr_account_id }}.dkr.ecr.${{ inputs.ecr_region }}.amazonaws.com" >> "$GITHUB_ENV"

        - name: Check if image exists in ECR
          id: check-image
          shell: bash
          run: |
              set -e  # Stop execution on any error

              IMAGE_INFO=$(aws ecr describe-images --repository-name "${{ inputs.service_name }}" --image-ids imageTag="${{ inputs.service_version }}" --region "${{ inputs.ecr_region }}" 2>&1 || echo "ERROR: $?")

              echo "$IMAGE_INFO" | tee aws_ecr_output.json  # Store response for debugging and print it

              if echo "$IMAGE_INFO" | grep -q "ImageNotFoundException"; then
                echo "🔴 ${{ inputs.service_name }}:${{ inputs.service_version }} does not exist in ECR. Proceeding with Docker build step"
                echo "IMAGE_EXISTS=false" >> "$GITHUB_OUTPUT"
              elif echo "$IMAGE_INFO" | grep -q "imageDetails"; then
                echo "✅ ${{ inputs.service_name }}:${{ inputs.service_version }} exists in ECR. Skipping Docker build step"
                echo "IMAGE_EXISTS=true" >> "$GITHUB_OUTPUT"
              else
                echo "⚠️ Unexpected error when checking ECR. See logs below:"
                echo "$IMAGE_INFO"
                exit 1  # Fail the job explicitly if an unexpected error occurs
              fi

        - name: Set up Docker Buildx
          uses: docker/setup-buildx-action@v2

        - name: Build and push Docker image
          uses: docker/build-push-action@v6
          if: steps.check-image.outputs.IMAGE_EXISTS == 'false'
          with:
              build-args: |
                  GITHUB_TOKEN=${{ env.GITHUB_TOKEN }}
                  NPM_TOKEN=${{ env.NPM_TOKEN }}
              context: ${{ inputs.context }}
              file: ${{ inputs.docker_file }}
              push: true
              tags: ${{ env.ECR_URI }}/${{ inputs.service_name }}:${{ inputs.service_version }}
              cache-from: type=registry,ref=${{ env.ECR_URI }}/buildcache:${{ inputs.service_name }}-buildcache
              cache-to: type=registry,mode=max,image-manifest=true,oci-mediatypes=true,ref=${{ env.ECR_URI }}/buildcache:${{ inputs.service_name }}-buildcache

Error I see

Error: buildx failed with: ERROR: failed to solve: failed to push <account_id>.dkr.ecr.<region>.amazonaws.com/<service_name>:5739d779ea29d5177694ed246b320ba4c51e4711: unexpected status from HEAD request to https://<account_id>.dkr.ecr.<region>.amazonaws.com/v2/<service_name>/blobs/sha256:f88d5d5eae58831962604a0d876619a86f23ffb127e798c2bbb9e2fe526ef785: 403 Forbidden

[i-am-gg]

@crazy-max can you please pitch in here for the help