Skip to content

[Feat][PD] lastest PD support from LMCache with NIXL #669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
+887 −204
Open

[Feat][PD] lastest PD support from LMCache with NIXL #669

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
1aa1feb
Custom modifications to production stack
kobe0938 Aug 27, 2025
b507357
try to pass CI
kobe0938 Aug 28, 2025
72141a0
pass pre-commit
kobe0938 Aug 28, 2025
7623b48
pass pre-commit
kobe0938 Aug 28, 2025
6b5a533
modify CI dockerfiles
kobe0938 Aug 28, 2025
099161f
wrap import NixlMsg into a try-catch block
kobe0938 Aug 28, 2025
b92826e
fuctional new docker image and remove nixlBackend UCX
kobe0938 Sep 5, 2025
1db19aa
only start the ZMQ task if the routing logic is RoutingLogic.DISAGGRE…
kobe0938 Sep 5, 2025
7998394
replace .github/values-10-disagg-prefill.yaml with values-16-disagg-p…
kobe0938 Sep 5, 2025
0b65402
pass pre-commit
kobe0938 Sep 5, 2025
4ebc430
fix yield
kobe0938 Sep 5, 2025
73f88a0
Merge remote-tracking branch 'upstream/main'
kobe0938 Sep 6, 2025
7dcc043
Change router tag
kobe0938 Sep 6, 2025
d64bafa
working version; temporarily push with older Dockerfile.pd
kobe0938 Sep 6, 2025
4fb4c01
temp opt working version
kobe0938 Sep 11, 2025
c98ab61
merge
kobe0938 Sep 11, 2025
faee188
fix
kobe0938 Sep 11, 2025
bdea203
CI fix for static router and k8s
kobe0938 Sep 11, 2025
9b2f57d
CI change
kobe0938 Sep 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 41 additions & 25 deletions .github/values-06-session-routing.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,70 +1,85 @@
# Unified configuration for disaggregated prefill setup
servingEngineSpec:
strategy:
type: Recreate
runtimeClassName: ""
enableEngine: true
runtimeClassName: "nvidia"
containerPort: 8000
modelSpec:
# Prefill node configuration
- name: "opt125m-prefill"
- name: "llama-prefill"
repository: "lmcache/vllm-openai"
tag: "2025-05-27-v1"
tag: "nightly-2025-09-04"
modelURL: "facebook/opt-125m"
replicaCount: 1
requestCPU: 8
requestMemory: "30Gi"
# requestGPU: 1
pvcStorage: "50Gi"
vllmConfig:
enablePrefixCaching: true
maxModelLen: 1024
v1: 1
gpuMemoryUtilization: 0.6
enablePrefixCaching: false
# maxModelLen: 2048
extraArgs:
- "--enforce-eager"
- "--disable-log-requests"
lmcacheConfig:
cudaVisibleDevices: "0"
enabled: true
kvRole: "kv_producer"
localCpu: true
maxLocalCpuSize: 5
maxLocalDiskSize: 0
enableNixl: true
enableXpyd: true
nixlRole: "sender"
nixlPeerHost: "vllm-opt125m-decode-engine-service"
nixlPeerPort: "55555"
nixlBufferSize: "1073741824" # 1GB
nixlProxyHost: "vllm-router-service"
nixlProxyPort: 7500
nixlBufferSize: "3774873600"
nixlBufferDevice: "cuda"
nixlEnableGc: true
enablePD: true
cpuOffloadingBufferSize: 0
rpcPort: "producer1"
labels:
model: "opt125m-prefill"
model: "llama-prefill"
chatTemplate: "chat.jinja2"
chatTemplateConfigMap: |-
{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}
{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{ '<|im_start|>assistant\n' }}{% endif %}
# hf_token: <hf-token>
# Decode node configuration
- name: "opt125m-decode"
- name: "llama-decode"
repository: "lmcache/vllm-openai"
tag: "2025-05-27-v1"
tag: "nightly-2025-09-04"
modelURL: "facebook/opt-125m"
replicaCount: 1
requestCPU: 8
requestMemory: "30Gi"
# requestGPU: 1
pvcStorage: "50Gi"
vllmConfig:
enablePrefixCaching: true
maxModelLen: 1024
v1: 1
enablePrefixCaching: false
# maxModelLen: 2048
extraArgs:
- "--enforce-eager"
- "--disable-log-requests"
lmcacheConfig:
cudaVisibleDevices: "1"
enabled: true
kvRole: "kv_consumer" # Set decode node as consumer
localCpu: false
maxLocalCpuSize: 0
enableNixl: true
enableXpyd: true
nixlRole: "receiver"
nixlPeerHost: "0.0.0.0"
nixlPeerPort: "55555"
nixlBufferSize: "1073741824" # 1GB
nixlPeerInitPort: 7300
nixlPeerAllocPort: 7400
nixlBufferSize: "3774873600"
nixlBufferDevice: "cuda"
nixlEnableGc: true
# nixlBackends: ["UCX"]
enablePD: true
rpcPort: "consumer1"
skipLastNTokens: 1
# hf_token: <hf-token>
labels:
model: "opt125m-decode"
model: "llama-decode"
chatTemplate: "chat.jinja2"
chatTemplateConfigMap: |-
{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}
Expand All @@ -75,7 +90,8 @@ servingEngineSpec:
- SYS_PTRACE

routerSpec:
repository: "git-act-router"
repository: "xiaokunchen/vllm-router"
tag: "09-10-v9"
imagePullPolicy: "IfNotPresent"
strategy:
type: Recreate
Expand Down
66 changes: 41 additions & 25 deletions .github/values-07-prefix-routing.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,70 +1,85 @@
# Unified configuration for disaggregated prefill setup
servingEngineSpec:
strategy:
type: Recreate
runtimeClassName: ""
enableEngine: true
runtimeClassName: "nvidia"
containerPort: 8000
modelSpec:
# Prefill node configuration
- name: "opt125m-prefill"
- name: "llama-prefill"
repository: "lmcache/vllm-openai"
tag: "2025-05-27-v1"
tag: "nightly-2025-09-04"
modelURL: "facebook/opt-125m"
replicaCount: 1
requestCPU: 8
requestMemory: "30Gi"
# requestGPU: 1
pvcStorage: "50Gi"
vllmConfig:
enablePrefixCaching: true
maxModelLen: 1024
v1: 1
gpuMemoryUtilization: 0.6
enablePrefixCaching: false
# maxModelLen: 2048
extraArgs:
- "--enforce-eager"
- "--disable-log-requests"
lmcacheConfig:
cudaVisibleDevices: "0"
enabled: true
kvRole: "kv_producer"
localCpu: true
maxLocalCpuSize: 5
maxLocalDiskSize: 0
enableNixl: true
enableXpyd: true
nixlRole: "sender"
nixlPeerHost: "vllm-opt125m-decode-engine-service"
nixlPeerPort: "55555"
nixlBufferSize: "1073741824" # 1GB
nixlProxyHost: "vllm-router-service"
nixlProxyPort: 7500
nixlBufferSize: "3774873600"
nixlBufferDevice: "cuda"
nixlEnableGc: true
enablePD: true
cpuOffloadingBufferSize: 0
rpcPort: "producer1"
labels:
model: "opt125m-prefill"
model: "llama-prefill"
chatTemplate: "chat.jinja2"
chatTemplateConfigMap: |-
{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}
{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{ '<|im_start|>assistant\n' }}{% endif %}
# hf_token: <hf-token>
# Decode node configuration
- name: "opt125m-decode"
- name: "llama-decode"
repository: "lmcache/vllm-openai"
tag: "2025-05-27-v1"
tag: "nightly-2025-09-04"
modelURL: "facebook/opt-125m"
replicaCount: 1
requestCPU: 8
requestMemory: "30Gi"
# requestGPU: 1
pvcStorage: "50Gi"
vllmConfig:
enablePrefixCaching: true
maxModelLen: 1024
v1: 1
enablePrefixCaching: false
# maxModelLen: 2048
extraArgs:
- "--enforce-eager"
- "--disable-log-requests"
lmcacheConfig:
cudaVisibleDevices: "1"
enabled: true
kvRole: "kv_consumer" # Set decode node as consumer
localCpu: false
maxLocalCpuSize: 0
enableNixl: true
enableXpyd: true
nixlRole: "receiver"
nixlPeerHost: "0.0.0.0"
nixlPeerPort: "55555"
nixlBufferSize: "1073741824" # 1GB
nixlPeerInitPort: 7300
nixlPeerAllocPort: 7400
nixlBufferSize: "3774873600"
nixlBufferDevice: "cuda"
nixlEnableGc: true
# nixlBackends: ["UCX"]
enablePD: true
rpcPort: "consumer1"
skipLastNTokens: 1
# hf_token: <hf-token>
labels:
model: "opt125m-decode"
model: "llama-decode"
chatTemplate: "chat.jinja2"
chatTemplateConfigMap: |-
{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}
Expand All @@ -75,7 +90,8 @@ servingEngineSpec:
- SYS_PTRACE

routerSpec:
repository: "git-act-router"
repository: "xiaokunchen/vllm-router"
tag: "09-10-v9"
imagePullPolicy: "IfNotPresent"
strategy:
type: Recreate
Expand Down
66 changes: 41 additions & 25 deletions .github/values-08-roundrobin-routing.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,70 +1,85 @@
# Unified configuration for disaggregated prefill setup
servingEngineSpec:
strategy:
type: Recreate
runtimeClassName: ""
enableEngine: true
runtimeClassName: "nvidia"
containerPort: 8000
modelSpec:
# Prefill node configuration
- name: "opt125m-prefill"
- name: "llama-prefill"
repository: "lmcache/vllm-openai"
tag: "2025-05-27-v1"
tag: "nightly-2025-09-04"
modelURL: "facebook/opt-125m"
replicaCount: 1
requestCPU: 8
requestMemory: "30Gi"
# requestGPU: 1
pvcStorage: "50Gi"
vllmConfig:
enablePrefixCaching: true
maxModelLen: 1024
v1: 1
gpuMemoryUtilization: 0.6
enablePrefixCaching: false
# maxModelLen: 2048
extraArgs:
- "--enforce-eager"
- "--disable-log-requests"
lmcacheConfig:
cudaVisibleDevices: "0"
enabled: true
kvRole: "kv_producer"
localCpu: true
maxLocalCpuSize: 5
maxLocalDiskSize: 0
enableNixl: true
enableXpyd: true
nixlRole: "sender"
nixlPeerHost: "vllm-opt125m-decode-engine-service"
nixlPeerPort: "55555"
nixlBufferSize: "1073741824" # 1GB
nixlProxyHost: "vllm-router-service"
nixlProxyPort: 7500
nixlBufferSize: "3774873600"
nixlBufferDevice: "cuda"
nixlEnableGc: true
enablePD: true
cpuOffloadingBufferSize: 0
rpcPort: "producer1"
labels:
model: "opt125m-prefill"
model: "llama-prefill"
chatTemplate: "chat.jinja2"
chatTemplateConfigMap: |-
{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}
{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{ '<|im_start|>assistant\n' }}{% endif %}
# hf_token: <hf-token>
# Decode node configuration
- name: "opt125m-decode"
- name: "llama-decode"
repository: "lmcache/vllm-openai"
tag: "2025-05-27-v1"
tag: "nightly-2025-09-04"
modelURL: "facebook/opt-125m"
replicaCount: 1
requestCPU: 8
requestMemory: "30Gi"
# requestGPU: 1
pvcStorage: "50Gi"
vllmConfig:
enablePrefixCaching: true
maxModelLen: 1024
v1: 1
enablePrefixCaching: false
# maxModelLen: 2048
extraArgs:
- "--enforce-eager"
- "--disable-log-requests"
lmcacheConfig:
cudaVisibleDevices: "1"
enabled: true
kvRole: "kv_consumer" # Set decode node as consumer
localCpu: false
maxLocalCpuSize: 0
enableNixl: true
enableXpyd: true
nixlRole: "receiver"
nixlPeerHost: "0.0.0.0"
nixlPeerPort: "55555"
nixlBufferSize: "1073741824" # 1GB
nixlPeerInitPort: 7300
nixlPeerAllocPort: 7400
nixlBufferSize: "3774873600"
nixlBufferDevice: "cuda"
nixlEnableGc: true
# nixlBackends: ["UCX"]
enablePD: true
rpcPort: "consumer1"
skipLastNTokens: 1
# hf_token: <hf-token>
labels:
model: "opt125m-decode"
model: "llama-decode"
chatTemplate: "chat.jinja2"
chatTemplateConfigMap: |-
{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}
Expand All @@ -75,7 +90,8 @@ servingEngineSpec:
- SYS_PTRACE

routerSpec:
repository: "git-act-router"
repository: "xiaokunchen/vllm-router"
Copy link
Contributor

@panpan0000 panpan0000 Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why change this line ?

Copy link
Contributor Author

@kobe0938 kobe0938 Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a temporary change to compose all the lasted change in this pr regarding the router.

They should be reverted in the end.

tag: "09-10-v9"
imagePullPolicy: "IfNotPresent"
strategy:
type: Recreate
Expand Down
Loading
Loading