[Fix] Fix FIA query and query_start_loc shape mismatch error
#4518
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request aims to fix a validation error in the FIA operator by adjusting the padding logic for queries in graph mode and multi-DP scenarios. The changes correctly generalize the padding logic and fix a bug in how sequence lengths and token locations were padded. However, I've identified a critical issue where the padding for block_table is inconsistent with the padding for seq_lens, which will likely lead to a shape mismatch and runtime errors. The proposed fix addresses this inconsistency.
Collaborator
|
Please check whether padding block_table is necessary and whether it causes potential excessive memory consumption. cc @yiz-liu |
weijinqian0
approved these changes
Nov 28, 2025
Collaborator
yiz-liu
changed the title
query and query_start_loc shape mismatch error
Collaborator
momo609
force-pushed
the
adde2e
branch
4 times, most recently
from
98841f5 to
abcb13f
Compare
MengqingCao
added
ready
momo609
force-pushed
the
adde2e
branch
3 times, most recently
from
8e02166 to
5b61368
Compare
Signed-off-by: wangxiaoxin-sherie <[email protected]>
yiz-liu
approved these changes
Dec 2, 2025
Mercykid-bash
pushed a commit
to Mercykid-bash/vllm-ascend
that referenced
this pull request
Dec 4, 2025
…m-project#4518) ### What this PR does / why we need it? Due to the requirement of the FIA operator that the **query.shape[0]** must match **actual_seq_len[-1]**, in graph mode and multi-DP scenarios, the query is padded to the size of **num_input_token**. This leads to validation errors during tiling in the operator. However, since the padding is applied at the end of the query, it does not affect the actual execution result of the operator, and the precision remains unaffected. <img width="2434" height="49" alt="image" src="https://github.com/user-attachments/assets/63520816-fbc3-4382-82b9-89dbb1492f6c" /> Our modification padding both **actual_seq_len** and **actual_seq_len_kv** to resolve the validation issue in the operator. ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.11.2 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.2 Signed-off-by: wangxiaoxin-sherie <[email protected]> Co-authored-by: wangxiaoxin-sherie <[email protected]> Signed-off-by: Che Ruan <[email protected]>
Mercykid-bash
pushed a commit
to Mercykid-bash/vllm-ascend
that referenced
this pull request
Dec 4, 2025
…m-project#4518) ### What this PR does / why we need it? Due to the requirement of the FIA operator that the **query.shape[0]** must match **actual_seq_len[-1]**, in graph mode and multi-DP scenarios, the query is padded to the size of **num_input_token**. This leads to validation errors during tiling in the operator. However, since the padding is applied at the end of the query, it does not affect the actual execution result of the operator, and the precision remains unaffected. <img width="2434" height="49" alt="image" src="https://github.com/user-attachments/assets/63520816-fbc3-4382-82b9-89dbb1492f6c" /> Our modification padding both **actual_seq_len** and **actual_seq_len_kv** to resolve the validation issue in the operator. ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - vLLM version: v0.11.2 - vLLM main: https://github.com/vllm-project/vllm/commit/v0.11.2 Signed-off-by: wangxiaoxin-sherie <[email protected]> Co-authored-by: wangxiaoxin-sherie <[email protected]> Signed-off-by: Che Ruan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it?
Due to the requirement of the FIA operator that the query.shape[0] must match actual_seq_len[-1], in graph mode and multi-DP scenarios, the query is padded to the size of num_input_token. This leads to validation errors during tiling in the operator. However, since the padding is applied at the end of the query, it does not affect the actual execution result of the operator, and the precision remains unaffected.
Our modification padding both actual_seq_len and actual_seq_len_kv to resolve the validation issue in the operator.
Does this PR introduce any user-facing change?
How was this patch tested?