k8s集群部署，不使用GPU进行推理

[Notailli]

System Info / 系統信息

cuda:12.8
python:3.11.7

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

• docker / docker
• pip install / 通过 pip install 安装
• installation from source / 从源码安装

Version info / 版本信息

xprobe/xinference:v1.11.0

The command used to start Xinference / 用以启动 xinference 的命令

具体步骤
新增 Xinference Helm 仓库

helm repo add xinference https://xorbitsai.github.io/xinference-helm-charts
更新仓库索引，查询可安装版本

helm repo update xinference
helm search repo xinference/xinference --devel --versions
安装

helm install xinference xinference/xinference -n xinference --version 0.0.1-v1.11.0

Reproduction / 复现过程

根据k8s部署之后，没有通过GPU进行模型推理，而是使用的CPU

Expected behavior / 期待表现

希望能够通过GPU进行模型推理

[Notailli]

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    meta.helm.sh/release-name: xinference
    meta.helm.sh/release-namespace: xinference
  creationTimestamp: "2025-10-29T07:50:09Z"
  generation: 2
  labels:
    app.kubernetes.io/instance: xinference
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: xinference
    app.kubernetes.io/version: v1.11.0
    helm.sh/chart: xinference-v1.11.0
  name: xinference-supervisor
  namespace: xinference
  resourceVersion: "122295135"
  uid: 79ba63e1-1407-4a15-ac47-601622a53643
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: xinference-supervisor
      app.kubernetes.io/instance: xinference
      app.kubernetes.io/name: xinference
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: xinference-supervisor
        app.kubernetes.io/instance: xinference
        app.kubernetes.io/name: xinference
    spec:
      containers:
      - args:
        - --port
        - "9997"
        - --host
        - $(POD_IP)
        - --supervisor-port
        - "9999"
        - --log-level
        - debug
        command:
        - xinference-supervisor
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: XINFERENCE_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: xprobe/xinference:v1.11.0
        imagePullPolicy: IfNotPresent
        name: supervisor
        ports:
        - containerPort: 9997
          name: web
          protocol: TCP
        - containerPort: 9999
          name: oscar
          protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: 4Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: "1"
            memory: 4Gi
            nvidia.com/gpu: "1"
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2025-10-29T08:34:15Z"
    lastUpdateTime: "2025-10-29T08:34:15Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2025-10-29T08:34:15Z"
    lastUpdateTime: "2025-10-30T01:33:21Z"
    message: ReplicaSet "xinference-supervisor-6899c4d469" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "9"
    meta.helm.sh/release-name: xinference
    meta.helm.sh/release-namespace: xinference
  creationTimestamp: "2025-10-29T07:50:09Z"
  generation: 23
  labels:
    app.kubernetes.io/instance: xinference
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: xinference
    app.kubernetes.io/version: v1.11.0
    helm.sh/chart: xinference-v1.11.0
  name: xinference-worker
  namespace: xinference
  resourceVersion: "122295730"
  uid: 53b6c3f9-0738-44a2-87a7-5f9db739175c
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: xinference-worker
      app.kubernetes.io/instance: xinference
      app.kubernetes.io/name: xinference
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: xinference-worker
        app.kubernetes.io/instance: xinference
        app.kubernetes.io/name: xinference
    spec:
      containers:
      - args:
        - -e
        - http://service-supervisor:9997
        - --host
        - $(POD_IP)
        - --worker-port
        - "30001"
        - --log-level
        - debug
        command:
        - xinference-worker
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: XINFERENCE_MODEL_SRC
          value: huggingface
        - name: XINFERENCE_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: xprobe/xinference:v1.11.0
        imagePullPolicy: IfNotPresent
        name: worker
        ports:
        - containerPort: 30001
          protocol: TCP
        resources:
          limits:
            cpu: "12"
            memory: 156Gi
            nvidia.com/gpu: "2"
          requests:
            cpu: "12"
            memory: 156Gi
            nvidia.com/gpu: "2"
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /models
          name: model-storage
        - mountPath: /workspace/outputs
          name: output-storage
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sh
        - -c
        - until curl -v http://service-supervisor:9997/v1/address; do echo waiting
          for supervisor; sleep 1; done
        image: curlimages/curl:8.8.0
        imagePullPolicy: IfNotPresent
        name: check-supervisor
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /data/Wan-AI
          type: Directory
        name: model-storage
      - hostPath:
          path: /data/Wan-AI/output
          type: Directory
        name: output-storage
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2025-10-30T01:34:19Z"
    lastUpdateTime: "2025-10-30T01:34:19Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2025-10-29T08:34:33Z"
    lastUpdateTime: "2025-10-30T01:34:19Z"
    message: ReplicaSet "xinference-worker-7b4775c5b6" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 23
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2

[Notailli]

[qinxuye]

看起来识别到了 GPU 呀。加载什么模型没有用到 GPU？

[Notailli]

是的，全程都没有用到GPU

[Notailli]

还想问一下wan2.2模型的分布式推理是一次性的么，还是推理之后，模型还在内存中，可提供下一次推理？

[qinxuye]

加载了模型之后可以进行下一次。