[Core] Encoder separation for Encode-Prefill-Decode Disaggregation #4176
Conversation
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for encode-prefill-decode disaggregation by separating the encoder execution path. Workers can now be designated as encoder producers, which exclusively run the multimodal encoder and transfer the resulting embeddings, bypassing the decoding process. The implementation correctly adds logic for producer and consumer roles. However, I've identified a critical bug where consumer ranks incorrectly re-execute the encoder, which overwrites the embeddings they just received. My review provides a necessary fix for this issue.
|
Are there any proxies that support EPD currently? |
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
amy-why-3459
force-pushed
the
epd_draft
branch
from
17f1100 to
63971b1
Compare
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
amy-why-3459
force-pushed
the
epd_draft
branch
from
63971b1 to
ad5aaa2
Compare
amy-why-3459
force-pushed
the
epd_draft
branch
6 times, most recently
from
170adfc to
073df5a
Compare
amy-why-3459
changed the title
wangxiyuan
added
ready
amy-why-3459
force-pushed
the
epd_draft
branch
4 times, most recently
from
92c5ecc to
adce4f1
Compare
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
Signed-off-by: amy-why-3459 <[email protected]>
amy-why-3459
force-pushed
the
epd_draft
branch
from
adce4f1 to
85306a9
Compare
|
Thanks for this great work, do you have any plan to add e2e test for this feature? |
We will add use cases as soon as possible. |
…llm-project#4176) ### What this PR does / why we need it? Support Encoder separation for Encode-Prefill-Decode Disaggregation - vLLM version: v0.11.2 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.2 Signed-off-by: amy-why-3459 <[email protected]> Signed-off-by: Che Ruan <[email protected]>
…llm-project#4176) ### What this PR does / why we need it? Support Encoder separation for Encode-Prefill-Decode Disaggregation - vLLM version: v0.11.2 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.2 Signed-off-by: amy-why-3459 <[email protected]> Signed-off-by: Che Ruan <[email protected]>
…llm-project#4176) ### What this PR does / why we need it? Support Encoder separation for Encode-Prefill-Decode Disaggregation - vLLM version: v0.11.2 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.2 Signed-off-by: amy-why-3459 <[email protected]>
4 participants
What this PR does / why we need it?
Encoder separation for Encode-Prefill-Decode Disaggregation
Does this PR introduce any user-facing change?
How was this patch tested?