GPU provisioning respected by SkyPilot job, but not isolated in SSH

[vim-hjk]

Environment

Server infra: Minikube Kubernetes
SkyPilot component tested: API Server deployment + Job execution
GPU hardware on node: NVIDIA RTX 4090 × 2

Description

When testing with the following SkyPilot task config, jobs work as expected:

• Job runs on exactly 1 GPU
• Kubernetes Pod resource limit is also set to 1 GPU (nvidia.com/gpu: 1)

Task Config Used (Quickstart example):

resources:
  # Optional; if left out, automatically pick the cheapest cloud.
  infra: aws
  accelerators: RTX4090:1

# Working directory (optional) containing the project codebase.
# Its contents are synced to ~/sky_workdir/ on the cluster.
workdir: .

# Typical use: pip install -r requirements.txt
# Invoked under the workdir (i.e., can use its files).
setup: |
  echo "Running setup."

# Typical use: make use of resources, such as running training.
# Invoked under the workdir (i.e., can use its files).
run: |
  echo "Hello, SkyPilot!"
  conda env list

Problem

Although job scheduling respects the GPU provisioning limit, isolation is not enforced at the user session or container runtime level:

• After connecting to the SkyPilot cluster via SSH, running nvidia-smi shows both GPUs (GPU0, GPU1).
• When executing a Python process inside the provisioned Pod container via SSH session, running DDP (Distributed Data Parallel) or other multi-GPU utilizing Python commands ends up using all 2 GPUs, not the 1 GPU provisioned or limited by SkyPilot. (Is this related to the NVIDIA GPU Operator?)

Has anyone encountered the same or a similar issue during local Kubernetes testing (like Minikube)?

[kyuds]

Thank you for raising the issue @vim-hjk ! The team will try to get to this soon.

Just some quick questions to help with fixing your issue:

• Am I right to assume your Minikube cluster runs on an on-prem server?
• Can you tell us which version of NVIDIA GPU Operator you have installed?

Note for rest of the SkyPilot team:
(related issues #2721 #3443)

Some basic triage:

• GPU isolation works when using GKE (sky launch -c mycluster --infra k8s --gpus T4 tested with k8s cluster with 4 nodes with 1 T4 each, and k8s cluster with 1 node with 2 T4s. nvidia-smi shows 1 GPU for both cases as expected. (Launching with --gpus T4:2 also works fine, shows 2 gpus in nvidia-smi).

[vim-hjk]

Thank you for the response.
The server infrastructure is an on-premises server environment,
the minikube version is v1.37.0
and the GPU Operator version is v25.10.0.

GPU driver version: 535.183.01

[kyuds]

@vim-hjk sorry for the late reply. We feel that this issue is related to the GPU operator installation, could you walk through our troubleshooting page and see if the logs show one GPU or two?