Commit 916a9a1
fix synchronize error of exceeds_max_model_len d2h copy (#4708)
### What this PR does / why we need it?
there is d2h copy blocking cpu operations in mtp propose method, which
make host bound issue. this pr refactor it and use cpu tensor to
implement it.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
vllm main f5d3d93c40417c296c20dc301100e55708a17f3f
- vLLM version: v0.12.0
- vLLM main:
vllm-project/vllm@ad32e3e
Signed-off-by: Ronald1995 <[email protected]>
Co-authored-by: wangxiyuan <[email protected]>1 parent 2be0fe2 commit 916a9a1
1 file changed
+3
-4
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
885 | 885 | | |
886 | 886 | | |
887 | 887 | | |
888 | | - | |
889 | | - | |
890 | | - | |
891 | | - | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
892 | 891 | | |
893 | 892 | | |
894 | 893 | | |
| |||
0 commit comments