[v0.11.0][Bugfix][MoE] enable force_load_balance in aclgraph #4367
Conversation
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enables force_load_balance for MoE layers when using ACL graph by removing a condition that previously disabled it. This allows torch.randint_like to be called within a graph context. My main concern is that this change contradicts an existing comment in the code which states that torch.randint_like is not supported in ACL graph, posing a critical risk of failure during profiling runs.
| # to avoid accumulating too much tokens on a single rank. | ||
| # currently it is only activated when doing profile runs. | ||
| if enable_force_load_balance and not self.use_aclgraph: | ||
| if enable_force_load_balance: |
There was a problem hiding this comment.
This change enables torch.randint_like to be executed within an ACL graph context. This contradicts the comment on lines 55-58, which states that torch.randint_like is unsupported in ACL graph and was intentionally disabled. Executing an unsupported operation within a graph will cause failures during profiling runs. This introduces a critical risk of breaking profiling with ACL graph.
Pr0Wh1teGivee
force-pushed
the
aclgraph_force_lb_011
branch
from
e0852e2 to
d62ec56
Compare
Pr0Wh1teGivee
force-pushed
the
aclgraph_force_lb_011
branch
2 times, most recently
from
1bcd112 to
a1bc7cd
Compare
weijinqian0
added
ready
Signed-off-by: Pr0Wh1teGivee <[email protected]>
Pr0Wh1teGivee
force-pushed
the
aclgraph_force_lb_011
branch
from
fe3cdfd to
242572d
Compare
|
This is a cherry-pick of #4366 |
…oject#4367) ### What this PR does / why we need it? Enable force_load_balance in aclgraph, solving OOM issues. pick from vllm-project#4366 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? e2e & ut Signed-off-by: Pr0Wh1teGivee <[email protected]> Signed-off-by: 刘哲续 <[email protected]>
…oject#4367) ### What this PR does / why we need it? Enable force_load_balance in aclgraph, solving OOM issues. pick from vllm-project#4366 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? e2e & ut Signed-off-by: Pr0Wh1teGivee <[email protected]>
5 participants
What this PR does / why we need it?
Enable force_load_balance in aclgraph, solving OOM issues.
Does this PR introduce any user-facing change?
No
How was this patch tested?
e2e & ut