Permission issues on docker swarm with NFS (Operation not permitted)

[vhaudiquet]

Hello!

I have an issue with recent mongodb images.

I have a setup that might be a bit weird: docker swarm deploys mongo with a volume that uses the local driver, that points to a folder that is actually a mounted NFS share. The stack file is as follows:

...
  mongo:
    hostname: mongo
    image: mongo:8.0.15
    restart: always
    user: 2000:2000 # this is recent, I tried with no user, user 999:999, root:root and 2000:2000
    volumes:
      - mongo_data:/data/db:Z
      - mongo_data:/data/configdb:Z
    env_file: .env
...

volumes:
  mongo_data:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/app/buildpath/data/_data'

/app is a mounted NFS share, to my NAS.

With mongo:8.0.15, the container starts and is healthy, but all write transactions fail. For example:

{"t":{"$date":"2025-12-06T13:16:20.040+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"conn28","msg":"createCollection","attr":{"namespace":"matches.15.24.1","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"76316150-f940-49f8-b319-a6290ce27a38"}},"options":{}}}
{"t":{"$date":"2025-12-06T13:16:20.047+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"conn28","msg":"WiredTiger error message","attr":{"error":1,"message":{"ts_sec":1765026980,"ts_usec":47630,"thread":"1:0x7f31565356c0","session_name":"WT_SESSION.create","category":"WT_VERB_DEFAULT","log_id":1000000,"category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_open_file:924:/data/db/collection-1-4269577862602222102.wt: handle-open: open","error_str":"Operation not permitted","error_code":1}}}

With the more recent image mongo:8.2.2, the container does not even start, as opening the WiredTiger database file fails with 'Operation not permitted', even in read mode.

I tried setting the user to root:root, 999:999 and 2000:2000. I tried changing the owner and permissions of /app/buildpath/data recursively to any of those, it does not seem to change anything.
My NFS share has the following options (server-side):

"/mnt/fast_app_data/docker-homeprod"\
        *(sec=sys,rw,anonuid=0,anongid=0,all_squash,insecure,no_subtree_check)

I'm using a mapall to root, so that might be a problem for mongo as well.

What I don't understand is that it used to work fine not too long ago, and those recent images seem to have changed something. I tried to look through existing issues but could not find anything recent.

Did something change recently? Is my use case not supported at all? Is that an already fixed problem (Sorry then! I could not find the solution through issues and forums)?

Thanks!

[vhaudiquet]

I guess one of the recent change could be d9efcb7 ?

I also noted that some of the files are being reset to root:root with my current config of having mongo run with 2000:2000:

...
-rwxrwxrwx 1 2000 2000 430182400 Nov 24 00:16 collection-0-1197853415329889252.wt
-rwxrwxrwx 1 2000 2000    348160 Nov 24 00:16 collection-0-12554627696672054276.wt
-rw------- 1 root root         0 Dec  6 13:43 collection-0-13988773120443256580.wt
-rwxrwxrwx 1 2000 2000 392589312 Nov 24 00:16 collection-0-14202213368659583766.wt
-rwxrwxrwx 1 2000 2000    196608 Nov 24 00:16 collection-0-1429850162317927748.wt
-rw------- 1 root root         0 Dec  6 11:12 collection-0-15014294931737174972.wt
-rwxrwxrwx 1 2000 2000 814022656 Nov 24 00:16 collection-0-15156747466538672499.wt
...

[tianon]

It can't be d9efcb7 because that entire block only runs if the container starts as root (and you're starting with user:). 😅

[vhaudiquet]

I guess the error with 8.0.15 could be that because of the mappall to root, new files created by the mongo container are given root:root owner. But still, I don't understand why then the mapall doesn't apply to allow mongo 2000:2000 to write that file as well...

[tianon]

I don't think we've changed anything in the image that could've caused this behavior, but I wouldn't be surprised if MongoDB-on-NFS is not something that's very well-supported or heavily-tested. 😬 🙈

[vhaudiquet]

Right, that does make a lot of sense. The thing that bothers me is that I also have a Kubernetes cluster that uses a mongo image, with csi-driver-nfs providing the persistentvolumeclaim, and I don't think I have the issue there.

I will try to investigate this, try to change the image on Kubernetes as well and debug a bit more, but I don't have a lot of time currently. I'll keep sending my updates on this discussion 😄