[Bug]: Non-Root image db migrations not working

[sj-rtl]

What happened?

Hello,

I noticed that at least with version 1.75.5 and 1.74.15 the non_root image cannot be used for database migrations any longer. When running this as part of of a Helm chart, rolled out via ArgoCD, I get the following permission error:

I ran almost the same yaml file as provided by LiteLLM: https://github.com/BerriAI/litellm/blob/main/deploy/charts/litellm-helm/templates/migrations-job.yaml

(had to make adjustments, as I had to add another container for Cloud SQL Proxy, as I'm running on GCP)

Interestingly, it should not try to run the migrations at all, as I'm running against a DB that is already established (or at least not all of them)

Relevant log output

2025-08-19 09:04:22,757 - litellm_proxy_extras - INFO - Running prisma migrate deploy
2025-08-19 09:04:25,444 - litellm_proxy_extras - INFO - prisma db error: prisma:warn Prisma doesn't know which engines to download for the Linux distro "wolfi". Falling back to Prisma engines built "debian".
Please report your experience by creating an issue at https://github.com/prisma/prisma/issues so we can add your distro to the list of known supported distros.
prisma:warn Prisma doesn't know which engines to download for the Linux distro "wolfi". Falling back to Prisma engines built "debian".
Please report your experience by creating an issue at https://github.com/prisma/prisma/issues so we can add your distro to the list of known supported distros.
Error: P3005
The database schema is not empty. Read more about how to baseline an existing production database: https://pris.ly/d/migrate-baseline
, e: Prisma schema loaded from schema.prisma
Datasource "client": PostgreSQL database "litellm", schema "public" at "localhost"
34 migrations found in prisma/migrations
2025-08-19 09:04:25,444 - litellm_proxy_extras - INFO - Database schema is not empty, creating baseline migration
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/litellm_proxy_extras/utils.py", line 257, in setup_database
    result = subprocess.run(
        ["prisma", "migrate", "deploy"],
    ...<3 lines>...
        text=True,
    )
  File "/usr/lib/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['prisma', 'migrate', 'deploy']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/app/litellm/proxy/prisma_migration.py", line 16, in <module>
    run_server(["--skip_server_startup"], standalone_mode=False)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.13/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/app/litellm/proxy/proxy_cli.py", line 780, in run_server
    PrismaManager.setup_database(use_migrate=not use_prisma_db_push)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/litellm/proxy/db/prisma_client.py", line 157, in setup_database
    return ProxyExtrasDBManager.setup_database(use_migrate=use_migrate)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/litellm_proxy_extras/utils.py", line 304, in setup_database
    ProxyExtrasDBManager._create_baseline_migration(schema_path)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/litellm_proxy_extras/utils.py", line 57, in _create_baseline_migration
    init_dir.mkdir(parents=True, exist_ok=True)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/pathlib/_local.py", line 722, in mkdir
    os.mkdir(self, mode)
    ~~~~~~~~^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.13/site-packages/litellm_proxy_extras/migrations/0_init'

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.75.5

Twitter / LinkedIn details

No response

[Ithanil]

#13848 fixes it for me. But tbh, I think the whole idea of writing into package directories and thus having some of them owned by nobody:nogroup is wrong.

[danielefranceschi]

 LiteLLM Proxy: LiteLLM Prisma Client Exception connect(): [Errno 13] Permission denied: '/root/.cache/prisma-python/binaries/5.4.2/ac9d7041ed77bcc8a8dbd2ab6616b39013829574/node_modules/prisma/query-engine-debian-openssl-3.0.x'

I've also tried to mount an emptydir into /root/.cache, but nothing.

My 2 cents: dependencies should be preinstalled in the container image.

[sj-rtl]

Yeah tbh I just "solved" this by running the migration job (I'm deploying LiteLLM with helm and have a job that runs for the migrations) once with the root image and then move back to the non_root image.
Afaik new migrations should just work, the issue is just creating the baseline I assume.

[Ithanil]

You could also use, for example, litellm/litellm-non_root:1.76.0.rc.1 , which contains #13848.

More info here: #14006 (comment)

[Elvincth]

Same issue in litellm-non_root:1.77.7-stable prisma migration job is not able to run correctly against a fresh DB

[Elvincth]

Full log

prisma:warn Prisma doesn't know which engines to download for the Linux distro "wolfi". Falling back to Prisma engines built "debian".
Please report your experience by creating an issue at https://github.com/prisma/prisma/issues so we can add your distro to the list of known supported distros.
prisma:warn Prisma doesn't know which engines to download for the Linux distro "wolfi". Falling back to Prisma engines built "debian".
Please report your experience by creating an issue at https://github.com/prisma/prisma/issues so we can add your distro to the list of known supported distros.
Prisma schema loaded from schema.prisma
Datasource "client": PostgreSQL database "litellm_db", schema "public" at "postgres:5432"

���  Your database is now in sync with your Prisma schema. Done in 189ms

Running generate... (Use --skip-generate to skip the generators)
�[2K�[1A�[2K�[GRunning generate... - Prisma Client Python (v0.11.0)

Some types are disabled by default due to being incompatible with Mypy, it is highly recommended
to use Pyright instead and configure Prisma Python to use recursive types. To re-enable certain types:

generator client {
  provider             = "prisma-client-py"
  recursive_type_depth = -1
}

If you need to use Mypy, you can also disable this message by explicitly setting the default value:

generator client {
  provider             = "prisma-client-py"
  recursive_type_depth = 5
}

For more information see: https://prisma-client-py.readthedocs.io/en/stable/reference/limitations/#default-type-limitations


�[2K�[1A�[2K�[GTraceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/prisma/generator/generator.py", line 1
12, in run
    self._on_request(request)
    ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/prisma/generator/generator.py", line 1
70, in _on_request
    self.generate(data)
    ~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/site-packages/prisma/generator/generator.py", line 2
55, in generate
    shutil.copy(data.schema_path, packaged_schema)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/shutil.py", line 428, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/shutil.py", line 262, in copyfile
    with open(dst, 'wb') as fdst:
         ~~~~^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.13/site-package
s/prisma/schema.prisma'

INFO:     Started server process [1]
INFO:     Waiting for application startup.

   �������     ������轁�����������������轁���������������轁������     �������     ���������   ���������
   �������     ������爗�尠�鐥�鐥�����婙�鐥�鐥�吲�����婙�鐥�鐥�鐥�鐥�吲������     �������     ����������� �����������
   �������     �������   �������   �������������  �������     �������     ������婙���������婙������
   �������     �������   �������   ������婙�鐥�鐥��  �������     �������     ������爗�尠�����婙�吲������
   ����������������轁������   �������   ����������������轁���������������轁���������������轁������ ��尠�鐥�� �������
   ��尠�鐥�鐥�鐥�鐥�鐥�鐥�吲�尠�鐥��   ��尠�鐥��   ��尠�鐥�鐥�鐥�鐥�鐥�鐥�吲�尠�鐥�鐥�鐥�鐥�鐥�鐥�吲�尠�鐥�鐥�鐥�鐥�鐥�鐥�吲�尠�鐥��     ��尠�鐥��

INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)

�[1;37m#------------------------------------------------------------#�[0m
�[1;37m#                                                            #�[0m
�[1;37m#               'A feature I really want is...'               #�[0m
�[1;37m#        https://github.com/BerriAI/litellm/issues/new        #�[0m
�[1;37m#                                                            #�[0m
�[1;37m#------------------------------------------------------------#�[0m

 Thank you for using LiteLLM! - Krrish & Ishaan



�[1;31mGive Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new�[0m


�[32mLiteLLM: Proxy initialized with Config, Set models:�[0m
�[32m    qwen3-next-80b-a3b-instruct�[0m
�[32m    qwen3-next-80b-a3b-instruct�[0m
�[32m    qwen3-next-80b-a3b-thinking�[0m
INFO:     10.128.4.1:37076 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.128.4.1:45474 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:45490 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.131.2.1:35260 - "GET /ui/model_hub_table/ HTTP/1.1" 404 Not Found
INFO:     10.131.0.1:58230 - "GET /ui/model_hub_table/ HTTP/1.1" 404 Not Found
INFO:     10.128.2.1:55344 - "GET /ui/model_hub_table/ HTTP/1.1" 404 Not Found
INFO:     10.129.2.1:36776 - "GET /ui/model_hub_table/ HTTP/1.1" 404 Not Found
INFO:     10.130.2.1:34732 - "GET / HTTP/1.1" 200 OK
INFO:     10.131.2.1:35272 - "GET /swagger/swagger-ui.css.map HTTP/1.1" 404 Not Found
INFO:     10.131.0.1:58236 - "GET /openapi.json HTTP/1.1" 200 OK
INFO:     10.130.2.1:34740 - "GET /ui/model_hub_table/ HTTP/1.1" 404 Not Found
INFO:     10.128.4.1:40074 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:40076 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.131.2.1:57944 - "GET /schedule/model_cost_map_reload/status HTTP/1.1" 200 OK
INFO:     10.128.4.1:57806 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:57814 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.128.4.1:51336 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:51346 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.128.4.1:60080 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:60086 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.128.4.1:39750 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:39736 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.129.2.1:60852 - "GET /schedule/model_cost_map_reload/status HTTP/1.1" 200 OK
INFO:     10.130.2.1:48918 - "GET /ui/model_hub_table/ HTTP/1.1" 404 Not Found
INFO:     10.131.2.1:58734 - "GET /ui/model_hub_table/ HTTP/1.1" 404 Not Found
INFO:     10.128.2.1:38548 - "GET /user/available_roles HTTP/1.1" 200 OK
INFO:     10.131.0.1:43152 - "GET /user/list?page=1&page_size=25&sort_by=created_at&sort_order=desc HTTP/1.1" 200 OK
INFO:     10.129.2.1:43778 - "GET /ui/model_hub_table/ HTTP/1.1" 404 Not Found
INFO:     10.128.4.1:50518 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:50516 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.128.4.1:50914 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:50916 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.128.4.1:33564 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:33554 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.128.4.1:60160 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:60148 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.131.0.1:44726 - "GET /schedule/model_cost_map_reload/status HTTP/1.1" 200 OK
INFO:     10.128.4.1:57918 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:57930 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.128.4.1:58132 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:58124 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.128.4.1:42526 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:42532 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.128.4.1:54106 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.128.4.1:54104 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.130.2.1:45822 - "GET /schedule/model_cost_map_reload/status HTTP/1.1" 200 OK

[lukasz-sroka-yohana]

Is there any update on this? We’re running into the same problem and would love to see it fixed.

[Elvincth]

Is there any update on this? We’re running into the same problem and would love to see it fixed.

I manage to get it run on my Openshift cluster using an older image litellm/litellm-non_root:1.76.0.rc.1

[wipash]

I have this working by setting:

• LITELLM_MIGRATION_DIR: /tmp/prisma
• USE_PRISMA_MIGRATE: "True"

and mounting an emptydir to /tmp/prisma

working on litellm-non_root:main-v1.79.1.rc.1

[dotmobo]

Hi, for me with openshift and the last non root image, migrations works but i have the following error in the end of the script :

OSError: [Errno 30] Read-only file system: '/usr/lib/python3.13/site-packages/prisma/schema.prisma'

The application starts and works.