[Bug]: Build Cache not utilized in Docker Compose Build

[fuerst-one]

Error Message and Logs

Since v432 or so (can't say exactly unfortunately), docker compose builds are not cached or the build cache not utilized anymore.
It could be coincidence, but it started when the way build- and runtime variables were injected changed and SOURCE_COMMIT was somehow added to my build as a fixed build arg.
Edit: As mentioned in the comments, COOLIFY_CONTAINER_NAME and COOLIFY_BUILD_SECRETS_HASH might also be culprits.

I use ${SOURCE_COMMIT} in another runtime variable, but not in build vars, so I would guess it does not have to be injected into the build. But it seems logical to me that something like this prevents the build from being cached.

As a workaround I already tried declaring the args manually in the Dockerfiles and/or compose.yaml or leave them out entirely, but SOURCE_COMMIT is always inserted automatically at the top - tested just by echoing directly after the first FROM.

Example Dockerfile

FROM ubuntu:24.04

LABEL org.opencontainers.image.authors="Montala Ltd"

ENV DEBIAN_FRONTEND="noninteractive"

RUN apt-get update && \
    apt-get install -y --fix-missing \
    nano \
    imagemagick \
    apache2 \
    subversion \
    ghostscript \
    antiword \
    poppler-utils \
    libimage-exiftool-perl \
    cron \
    postfix \
    wget \
    curl \
    php \
    php-apcu \
    php-curl \
    php-dev \
    php-gd \
    php-intl \
    php-mysqlnd \
    php-mbstring \
    php-zip \
    libapache2-mod-php \
    ffmpeg \
    libopencv-dev \
    python3-opencv \
    python3 \
    python3-pip \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*

# Increase default upload limits
RUN sed -i -e "s/upload_max_filesize\s*=\s*2M/upload_max_filesize = 100M/g" /etc/php/8.3/apache2/php.ini \
 && sed -i -e "s/post_max_size\s*=\s*8M/post_max_size = 100M/g" /etc/php/8.3/apache2/php.ini \
 && sed -i -e "s/max_execution_time\s*=\s*30/max_execution_time = 300/g" /etc/php/8.3/apache2/php.ini \
 && sed -i -e "s/memory_limit\s*=\s*128M/memory_limit = 1G/g" /etc/php/8.3/apache2/php.ini

# Allow Apache to serve files from the web root
RUN printf '<Directory /var/www/>\n\
\tOptions FollowSymLinks\n\
</Directory>\n'\
>> /etc/apache2/sites-enabled/000-default.conf

# Allow fonts to be loaded from other domains
RUN a2enmod headers
RUN printf '\n<FilesMatch ".(ttf|otf|eot|woff|woff2)$">\n\
\tHeader set Access-Control-Allow-Origin "*"\n\
</FilesMatch>\n'\
>> /etc/apache2/apache2.conf
RUN apachectl -k graceful

# Add the hitcount cronjob to the daily cronjobs
ADD /containers/resourcespace/cronjob /etc/cron.daily/resourcespace

WORKDIR /var/www/html

# Checkout the latest version of ResourceSpace and prepare the directory structure
RUN rm -f index.html \
 && svn co -q https://svn.resourcespace.com/svn/rs/releases/10.5 . \
 && mkdir -p filestore \
 && chmod 777 filestore \
 && chmod -R 777 include/

# Copy the plugins and healthcheck route
COPY /containers/resourcespace/plugins/configurator_asset_config /var/www/html/plugins/configurator_asset_config
COPY /containers/resourcespace/plugins/configurator_file_editor /var/www/html/plugins/configurator_file_editor
COPY /containers/resourcespace/pages/healthcheck.php /var/www/html/pages

# Add healthcheck
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
    CMD curl -f http://localhost/pages/healthcheck.php || exit 1

# Also tested to declare ARG SOURCE_COMMIT here, but it was still auto-inserted at the top
# ARG SOURCE_COMMIT

CMD apachectl -D FOREGROUND

Example docker-compose.yaml

services:
  resourcespace:
    build: 
      context: .
      dockerfile: containers/resourcespace/Dockerfile.coolify
    restart: unless-stopped
    depends_on:
      resourcespace_db:
        condition: service_healthy
    ports:
      - "${RESOURCESPACE_PORT}:80"
    volumes:
      - resourcespace_filestore:/var/www/html/filestore
      - resourcespace_include:/var/www/html/include
  
  resourcespace_db:
    image: mysql/mysql-server:5.7
    environment:
      - MYSQL_USER=${RESOURCESPACE_DB_USERNAME}
      - MYSQL_PASSWORD=${RESOURCESPACE_DB_PASSWORD}
      - MYSQL_DATABASE=${RESOURCESPACE_DB_DATABASE}
      - MYSQL_TCP_PORT=${RESOURCESPACE_DB_PORT}
    ports:
      - '${RESOURCESPACE_DB_PORT}:${RESOURCESPACE_DB_PORT}'
    volumes:
      - 'resourcespace_db_data:/var/lib/mysql'
...

Logs:

Starting deployment of healthy-hummingbird-nc08so4c0k4808w04gcc0kcs to localhost.
2025-Oct-28 09:10:50.791761
Preparing container with helper image: ghcr.io/coollabsio/coolify-helper:1.0.11
2025-Oct-28 09:10:50.918138
[CMD]: docker stop --time=30 fsccogs4swgcosk4oss40gw4
2025-Oct-28 09:10:50.918138
Flag --time has been deprecated, use --timeout instead
2025-Oct-28 09:10:50.923398
Error response from daemon: No such container: fsccogs4swgcosk4oss40gw4
2025-Oct-28 09:10:51.028243
[CMD]: docker rm -f fsccogs4swgcosk4oss40gw4
2025-Oct-28 09:10:51.028243
Error response from daemon: No such container: fsccogs4swgcosk4oss40gw4
2025-Oct-28 09:10:51.205609
[CMD]: docker run -d --network coolify --name fsccogs4swgcosk4oss40gw4  --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/coollabsio/coolify-helper:1.0.11
2025-Oct-28 09:10:51.205609
0a591216bc201282c865b7be30dda2fb3ed5388fb5b46c9ec7b86cbd443c3026
2025-Oct-28 09:10:53.805776
[CMD]: docker exec fsccogs4swgcosk4oss40gw4 bash -c 'GIT_SSH_COMMAND="ssh -o ConnectTimeout=30 -p 22 -o Port=22 -o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git ls-remote git@github.com:...'
2025-Oct-28 09:10:53.805776
2a7fdb0bf343357109bfa851dfbf75a27b182128	refs/heads/staging
2025-Oct-28 09:10:53.921876
----------------------------------------
2025-Oct-28 09:10:53.933903
Importing git@github.com:...
[CMD]: docker exec fsccogs4swgcosk4oss40gw4 bash -c 'mkdir -p /root/.ssh' && docker exec fsccogs4swgcosk4oss40gw4 bash -c 'echo '....' | base64 -d | tee /root/.ssh/id_rsa > /dev/null' && docker exec fsccogs4swgcosk4oss40gw4 bash -c 'chmod 600 /root/.ssh/id_rsa' && docker exec fsccogs4swgcosk4oss40gw4 bash -c 'GIT_SSH_COMMAND="ssh -o ConnectTimeout=30 -p 22 -o Port=22 -o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /root/.ssh/id_rsa" git clone --depth=1 --recurse-submodules --shallow-submodules -b 'staging' 'git@github.com:...' '/artifacts/fsccogs4swgcosk4oss40gw4' && cd '/artifacts/fsccogs4swgcosk4oss40gw4' && if [ -f .gitmodules ]; then git submodule sync && GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git submodule update --init --recursive --depth=1; fi && cd '/artifacts/fsccogs4swgcosk4oss40gw4' && GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git lfs pull'
[CMD]: docker exec fsccogs4swgcosk4oss40gw4 bash -c 'cd /artifacts/fsccogs4swgcosk4oss40gw4 && git log -1 some-commit-sha --pretty=%B'
2025-Oct-28 09:10:58.651993
Commit Message
2025-Oct-28 09:11:04.108079
[CMD]: docker exec fsccogs4swgcosk4oss40gw4 bash -c 'test -f /artifacts/fsccogs4swgcosk4oss40gw4/Dockerfile.coolify && echo 'exists' || echo 'not found''
2025-Oct-28 09:11:04.108079
exists
2025-Oct-28 09:11:04.247529
[CMD]: docker exec fsccogs4swgcosk4oss40gw4 bash -c 'cat /artifacts/fsccogs4swgcosk4oss40gw4/Dockerfile.coolify'
2025-Oct-28 09:10:54.444774
Cloning into '/artifacts/fsccogs4swgcosk4oss40gw4'...
...dockerfile contents...
2025-Oct-28 09:01:16.120414
Added 31 ARG declarations to Dockerfile for service app.
...dockerfile contents...
2025-Oct-28 08:21:23.131395
Pulling & building required images.
2025-Oct-28 08:21:23.185471
Creating build-time .env file in /artifacts (outside Docker context).
2025-Oct-28 08:21:23.913417
[CMD]: docker exec rsog0kck08ok8g0k8ow0wkcc bash -c 'cat /artifacts/build-time.env'
2025-Oct-28 08:21:23.913417
SOURCE_COMMIT='some-commit-sha'
...other variables...
2025-Oct-28 08:21:23.919504
Adding build arguments to Docker Compose build command.
2025-Oct-28 08:21:24.417601
[CMD]: docker exec rsog0kck08ok8g0k8ow0wkcc bash -c 'SOURCE_COMMIT=some-commit-sha COOLIFY_BRANCH=staging COOLIFY_RESOURCE_UUID=nc08so4c0k4808w04gcc0kcs COOLIFY_CONTAINER_NAME=nc08so4c0k4808w04gcc0kcs-082058501847  docker compose --env-file /artifacts/build-time.env --project-name nc08so4c0k4808w04gcc0kcs --project-directory /artifacts/rsog0kck08ok8g0k8ow0wkcc -f /artifacts/rsog0kck08ok8g0k8ow0wkcc/docker-compose-coolify.yml build --pull --build-arg SOURCE_COMMIT --build-arg COOLIFY_URL --build-arg COOLIFY_FQDN --build-arg APP_PORT --build-arg DB_DATABASE --build-arg DB_HOST --build-arg DB_PASSWORD --build-arg DB_PORT --build-arg DB_USERNAME --build-arg COOLIFY_BUILD_SECRETS_HASH=...'

...build logs...


2025-Oct-28 09:16:25.134292
[CMD]: docker stop --time=30 resourcespace-nc08so4c0k4808w04gcc0kcs-090115283376
2025-Oct-28 09:16:25.134292
Flag --time has been deprecated, use --timeout instead
2025-Oct-28 09:16:55.543555
resourcespace-nc08so4c0k4808w04gcc0kcs-090115283376
...
Starting new application.
2025-Oct-28 09:17:03.491858
[CMD]: docker exec fsccogs4swgcosk4oss40gw4 bash -c 'SOURCE_COMMIT=2a7fdb0bf343357109bfa851dfbf75a27b182128 COOLIFY_BRANCH=staging COOLIFY_RESOURCE_UUID=nc08so4c0k4808w04gcc0kcs COOLIFY_CONTAINER_NAME=nc08so4c0k4808w04gcc0kcs-091049327807  docker compose --env-file /artifacts/fsccogs4swgcosk4oss40gw4/.env --project-name nc08so4c0k4808w04gcc0kcs --project-directory /artifacts/fsccogs4swgcosk4oss40gw4 -f /artifacts/fsccogs4swgcosk4oss40gw4/docker-compose-coolify.yml up -d'

2025-Oct-28 09:17:34.590327
Container resourcespace-nc08so4c0k4808w04gcc0kcs-091103681295  Starting
2025-Oct-28 09:17:34.848362
Container resourcespace-nc08so4c0k4808w04gcc0kcs-091103681295  Started

2025-Oct-28 09:17:36.090641
New container started.
2025-Oct-28 09:17:37.076947
Gracefully shutting down build container: fsccogs4swgcosk4oss40gw4
2025-Oct-28 09:17:37.351147
[CMD]: docker stop --time=30 fsccogs4swgcosk4oss40gw4
2025-Oct-28 09:17:37.351147
Flag --time has been deprecated, use --timeout instead
2025-Oct-28 09:17:37.516748
fsccogs4swgcosk4oss40gw4
2025-Oct-28 09:17:37.762308
[CMD]: docker rm -f fsccogs4swgcosk4oss40gw4
2025-Oct-28 09:17:37.762308
Error response from daemon: No such container: fsccogs4swgcosk4oss40gw4

Steps to Reproduce

1. Disable "Force Docker Cleanup" in Server Settings
2. Add enough Docker Cleanup schedule headroom to fully build at least 2 times
3. Have more available disk space than in Docker cleanup threshold
4. Create resource using docker compose build
5. Create setup similar to description
6. (Maybe: Use SOURCE_COMMIT in runtime variable)
7. Deploy to init build cache
8. Deploy again
   => no build cache used, builds from scratch again (except docker images)

Example Repository URL

No response

Coolify Version

v4.0.0-beta.438

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubuntu 24.04.3 LTS

Additional Information

No response

Linear Issue: COOLGH-85

[djsisson]

@fuerst-one one of the build args is this:

COOLIFY_CONTAINER_NAME=nc08so4c0k4808w04gcc0kcs-082058501847

this will change on every run (which also seems to change from build to run in the above logs).

Therefore nothing will ever get cached, as this as well as the secrets hash will always be different on each run, and coolify (wrongly imo) insists on adding build args and secrets to a Dockerfile itself starting at the top

[fuerst-one]

Hey @djsisson thanks for the reply. I agree, that's probably the same issue, variables injected into build, that are effectively preventing cache. At least that's what I understand. Will add it to the description 👍

[pkpio]

It definitely started around the time how variables were tagged or injected. I had done lots of optimisations for my build speed using docker layers just a few weeks prior. Sad that we cannot use the layer caching in docker anymore.

[djsisson]

tbh i've no idea why a service needs to know it's own container name, and it's surely broken in compose setups anyway as there is multiple containers but only one arg, a quick fix would be to remove this env from coolify as i can't see it being needed, as other services would use the service names anyway not container names.

[pkpio]

Can confirm issue still occurs without ever using container name or source commit in the code (not in build time or run time).

[pkpio]

I think this commit caused the bug - acf2d5f (author: @andrasbacsai )

[pkpio]

I hate to spam but this has become a serious pain for our day to day workflow. Our cache optimised builds take about 2-5 minutes to deploy but now each build takes over 30 - 50 minutes as all layers built every time.

I hope a fix can be made available sooner. Happy to contribute if someone can share tips on where to look.