CSI not sending secrets as part of NodeStageVolume call

[Lol3rrr]

Nomad version

Nomad v1.10.5
BuildDate 2025-09-09T14:36:45Z
Revision a3b86c697f38ab032e1acaae8503ed10815bc4a2

Operating system and Environment details

Nodes are running Ubuntu Server 22.04
3x Clients and 3x Servers collocated on the same servers

Issue

I am using the ceph-csi plugin to provision CSI volumes, recently it started that the NodeStageVolume grpc call failed. The ceph-csi plugin claims to not receive the configured secrets and therefore cannot perform the operation.

For this I also forked the ceph-csi plugin to add more logging, which seems to confirm this idea.

Reproduction steps

1. Setup the ceph-csi plugin
2. Create a CSI volume using the provided file
3. Try to use it as part of a job

Expected Result

The CSI volume should mount normally

Actual Result

The CSI pre_hook fails

Log output from my modified ceph-csi version showing that there is no Secret send as part of the request

I0914 15:32:29.218823       1 utils.go:347] ID: 8 Req-ID: 0001-0024-8b83d68a-0635-11ee-94c5-29c911c40e53-0000000000000004-f1f1f6b8-4f43-4a7a-979a-f8b500718107 GRPC call: /csi.v1.Node/NodeStageVolume
I0914 15:32:29.220128       1 utils.go:348] ID: 8 Req-ID: 0001-0024-8b83d68a-0635-11ee-94c5-29c911c40e53-0000000000000004-f1f1f6b8-4f43-4a7a-979a-f8b500718107 GRPC request: {"staging_target_path":"/local/csi/staging/default/mc-volume-claudios/rw-file-system-single-node-writer","volume_capability":{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4"}},"volume_context":{"clusterID":"8b83d68a-0635-11ee-94c5-29c911c40e53","imageFeatures":"layering","imageName":"csi-vol-f1f1f6b8-4f43-4a7a-979a-f8b500718107","journalPool":"test-rbd","pool":"test-rbd"},"volume_id":"0001-0024-8b83d68a-0635-11ee-94c5-29c911c40e53-0000000000000004-f1f1f6b8-4f43-4a7a-979a-f8b500718107"}
E0914 15:32:29.220383       1 nodeserver.go:350] ID: 8 Req-ID: 0001-0024-8b83d68a-0635-11ee-94c5-29c911c40e53-0000000000000004-f1f1f6b8-4f43-4a7a-979a-f8b500718107 Entered NodeStageVolume Handler with request: volume_id:"0001-0024-8b83d68a-0635-11ee-94c5-29c911c40e53-0000000000000004-f1f1f6b8-4f43-4a7a-979a-f8b500718107" staging_target_path:"/local/csi/staging/default/mc-volume-claudios/rw-file-system-single-node-writer" volume_capability:{mount:{fs_type:"ext4"} access_mode:{mode:SINGLE_NODE_WRITER}} volume_context:{key:"clusterID" value:"8b83d68a-0635-11ee-94c5-29c911c40e53"} volume_context:{key:"imageFeatures" value:"layering"} volume_context:{key:"imageName" value:"csi-vol-f1f1f6b8-4f43-4a7a-979a-f8b500718107"} volume_context:{key:"journalPool" value:"test-rbd"} volume_context:{key:"pool" value:"test-rbd"}
E0914 15:32:29.220420       1 nodeserver.go:354] ID: 8 Req-ID: 0001-0024-8b83d68a-0635-11ee-94c5-29c911c40e53-0000000000000004-f1f1f6b8-4f43-4a7a-979a-f8b500718107 Failed to validate Request
E0914 15:32:29.220445       1 utils.go:352] ID: 8 Req-ID: 0001-0024-8b83d68a-0635-11ee-94c5-29c911c40e53-0000000000000004-f1f1f6b8-4f43-4a7a-979a-f8b500718107 GRPC error: rpc error: code = InvalidArgument desc = stage secrets cannot be nil or empty

Volume File

id           = "traefik"
name         = "traefik"
type         = "csi"
plugin_id    = "ceph-csi"
capacity_max = "10G"
capacity_min = "1G"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

secrets {
  userID  = "--REDACTED--"
  userKey = "--REDACTED--"
}

parameters {
  clusterID     = "8b83d68a-0635-11ee-94c5-29c911c40e53"
  pool          = "test-rbd"
  imageFeatures = "layering"
}

[tgross]

Hi @Lol3rrr! I took a quick look at this and confirm that we're not threading that down to the NodeStageVolume request (or the NodePublishVolume request either, but you're not getting that far and/or maybe your plugin doesn't need it).

This looks like a pretty straightforward fix as I think the client is getting the secrets from the server and then just ignoring them (🤦). I'll try to confirm that and then see if I can work up a patch later this week.

(Internal ref https://hashicorp.atlassian.net/browse/NMD-991)

[tgross]

Hi again @Lol3rrr! I've had a second look at this and I'm having trouble reproducing what you're seeing.

My earlier thought that we weren't threading the secrets to the NodePublishRequest didn't hold up on closer examination. We pass the whole volume to manager.MountVolume and then we extract those for the NodeStageVolumeRequest here and the NodePublishVolume call here.

I've further verified that a plugin is getting the expected values by using the hostpath plugin demo, where I get the following logs in the plugin (see the secret "xyzzy"):

I0917 17:40:56.132729       1 server.go:159] gRPCCall: {"Method":"/csi.v1.Node/NodeStageVolume","Request":{"volume_id":"76f86190-93ed-11f0-8754-3eb99c7cd846","staging_target_path":"/local/csi/staging/default/test-volume[1]/ro-file-system-single-node-reader-only","volume_capability":{"AccessType":{"Mount":{"mount_flags":["ro"]}},"access_mode":{"mode":2}},"secrets":{"somesecret":"xyzzy"}},"Response":{},"Error":"","FullError":null}
...
I0917 17:40:56.147101       1 server.go:159] gRPCCall: {"Method":"/csi.v1.Node/NodePublishVolume","Request":{"volume_id":"76f4ea77-93ed-11f0-8754-3eb99c7cd846","staging_target_path":"/local/csi/staging/default/test-volume[0]/ro-file-system-single-node-reader-only","target_path":"/local/csi/per-alloc/8422d18a-6177-39c9-a08b-6d110821e1a8/test-volume[0]/ro-file-system-single-node-reader-only","volume_capability":{"AccessType":{"Mount":{"mount_flags":["ro"]}},"access_mode":{"mode":2}},"readonly":true,"secrets":{"somesecret":"xyzzy"}},"Response":{},"Error":"","FullError":null}

Unfortunately I haven't been able to reproduce on Ceph because our Ceph demo hasn't been updated in a while and is giving me trouble (creating the volume is hanging in the Ceph container). Do you have more detail on the setup you're using?

[Lol3rrr]

Hi @tgross,
first off, thanks for the response and looking into the issue.
I am not sure what more details would be of use to you right now, but I can take a look how I could maybe reconstruct a smaller example that reproduces the issue.

This is also what threw me off at the start, as it was previously working and then suddenly stopped working.
As this issue seems very specific to my current setup/cluster, I was thinking about looking into it myself and seeing if maybe the secrets are not properly stored/forwarded/whatever in nomad itself (like some form of corruption or data loss). Do you happen to know any good starting points at which I could dump this sort of information to try and find where the secrets are "lost"?

[tgross]

This is tricky because we do everything possible to avoid leaking secrets in the API or logs!

If you take a snapshot via nomad operator snapshot save, the resulting file will contain unredacted volume info including the secrets. You can then read the snapshot via nomad operator snapshot state ./$filename.snap | jq '.CSIVolumes'.

(Note this snapshot file is a full cluster backup and contains the cluster keyring's decryption key unless you're using one of the keyring providers like "awskms" or Vault "transit", so be sure to dispose of the snapshot properly when you're done.)

as it was previously working and then suddenly stopped working.

Any chance the version of Ceph or the Ceph plugin changed in between it working and not working?

[Lol3rrr]

I just checked all the relevant version updates and I cannot find a correlation, as the only ceph related updates I performed were after the issues already started for some volumes or way before the issues started.

However based on the snapshot of the cluster, I noticed that some of the CSI volumes no longer have secrets set for them.
All of them were created using the same terraform configuration and are all still managed using terraform, so I am not sure why they suddenly lost their secrets

[tgross]

All of them were created using the same terraform configuration and are all still managed using terraform, so I am not sure why they suddenly lost their secrets

Oh, that's an interesting clue! Maybe there's a Terraform provider issue here where the redaction is making its way back into the TF state and that's wiping out the secret. I'll see if I can reproduce.