Skip to content

[Bugfix] Fix encoder-only model support for transformers backend #28021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Nov 5, 2025
Merged

[Bugfix] Fix encoder-only model support for transformers backend #28021

merged 3 commits into from
Nov 5, 2025
+16 −10

Conversation

@Isotr0py
Copy link
Member

@Isotr0py Isotr0py commented Nov 4, 2025 •
edited by github-actions bot
Loading

Purpose

(EngineCore_DP0 pid=5260) INFO 11-04 03:34:52 [base.py:120] Using Transformers backend.
(EngineCore_DP0 pid=5260) INFO 11-04 03:34:52 [cuda.py:414] Using FlexAttention backend.
(EngineCore_DP0 pid=5260) [2025-11-04 03:34:54] INFO _client.py:1025: HTTP Request: GET https://huggingface.co/api/models/papluca/xlm-roberta-base-language-detection "HTTP/1.1 200 OK"
(EngineCore_DP0 pid=5260) [2025-11-04 03:34:54] INFO _client.py:1025: HTTP Request: GET https://huggingface.co/api/models/papluca/xlm-roberta-base-language-detection/tree/main?recursive=false&expand=false "HTTP/1.1 200 OK"
(EngineCore_DP0 pid=5260) [2025-11-04 03:34:54] INFO _client.py:1025: HTTP Request: GET https://huggingface.co/api/models/papluca/xlm-roberta-base-language-detection/revision/main "HTTP/1.1 200 OK"
(EngineCore_DP0 pid=5260) [2025-11-04 03:34:55] INFO _client.py:1025: HTTP Request: HEAD https://huggingface.co/papluca/xlm-roberta-base-language-detection/resolve/main/model.safetensors.index.json "HTTP/1.1 404 Not Found"
(EngineCore_DP0 pid=5260) INFO 11-04 03:34:55 [weight_utils.py:480] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.94it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.93it/s]
(EngineCore_DP0 pid=5260) 
(EngineCore_DP0 pid=5260) INFO 11-04 03:34:55 [default_loader.py:314] Loading weights took 0.65 seconds
(EngineCore_DP0 pid=5260) INFO 11-04 03:34:56 [gpu_model_runner.py:2997] Model loading took 0.5192 GiB and 3.491527 seconds
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843] EngineCore failed to start.
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843] Traceback (most recent call last):
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]   File "/kaggle/working/vllm/vllm/v1/engine/core.py", line 834, in run_engine_core
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]   File "/kaggle/working/vllm/vllm/v1/engine/core.py", line 602, in __init__
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]     super().__init__(
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]   File "/kaggle/working/vllm/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]   File "/kaggle/working/vllm/vllm/v1/engine/core.py", line 223, in _initialize_kv_caches
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]     kv_cache_specs = self.model_executor.get_kv_cache_specs()
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]   File "/kaggle/working/vllm/vllm/v1/executor/abstract.py", line 129, in get_kv_cache_specs
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]     return self.collective_rpc("get_kv_cache_spec")
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]   File "/kaggle/working/vllm/vllm/v1/executor/uniproc_executor.py", line 73, in collective_rpc
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]   File "/kaggle/working/vllm/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]     return func(*args, **kwargs)
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]   File "/kaggle/working/vllm/vllm/v1/worker/gpu_worker.py", line 373, in get_kv_cache_spec
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]     return self.model_runner.get_kv_cache_spec()
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]   File "/kaggle/working/vllm/vllm/v1/worker/gpu_model_runner.py", line 4746, in get_kv_cache_spec
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]     if spec := attn_module.get_kv_cache_spec(self.vllm_config):
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]   File "/kaggle/working/vllm/vllm/attention/layer.py", line 462, in get_kv_cache_spec
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]     assert self.attn_type == AttentionType.DECODER
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=5260) ERROR 11-04 03:34:56 [core.py:843] AssertionError

Test Plan

pytest -s -v tests/models/test_initialization.py -k TransformersForSequenceClassification

Test Result

Test should pass with nightly Transformers now

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@Isotr0py Isotr0py requested a review from hmellor as a code owner November 4, 2025 05:22
gemini-code-assist[bot]
gemini-code-assist bot reviewed Nov 4, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a bug that prevented encoder-only models from working with the transformers backend. The issue, as indicated by the traceback, was an assertion error during the KV cache initialization, where the attention type was expected to be DECODER. The fix involves conditionally selecting the EncoderOnlyAttention class for encoder layers, which correctly signals that no KV cache is needed. The change is precise, correct, and effectively resolves the reported bug.

@heheda12345 heheda12345 enabled auto-merge (squash) November 4, 2025 05:39
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 4, 2025
DarkLight1337
DarkLight1337 reviewed Nov 4, 2025
View reviewed changes
vllm/model_executor/models/transformers/base.py
if attn_type == AttentionType.ENCODER_ONLY
else Attention
)
attention_instances[i] = attn_cls(
Copy link
Member

@DarkLight1337 DarkLight1337 Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does passing attn_type not work? Are the two not equivalent?

Copy link
Member

@hmellor hmellor Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#26587 (comment)

NickLucche
NickLucche reviewed Nov 4, 2025
View reviewed changes
vllm/model_executor/models/transformers/base.py
per_layer_sliding_window = self.config.sliding_window

attention_instances[i] = Attention(
attn_cls = (
Copy link
Collaborator

@NickLucche NickLucche Nov 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@heheda12345 do you want to handle it inside the Attention class init to signal deprecation with a warning?

Copy link
Collaborator

@heheda12345 heheda12345 Nov 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to handle it like this now. For the deprecation warning, just add one line of warning in Attention class? (not necessary in this PR)

@Isotr0py Isotr0py requested a review from ywang96 as a code owner November 4, 2025 16:49
@vllm-bot vllm-bot merged commit 0ff05e3 into vllm-project:main Nov 5, 2025
51 of 53 checks passed
@github-project-automation github-project-automation bot moved this from Todo to Done in Transformers backend Nov 5, 2025
@Isotr0py Isotr0py deleted the fix-transformers-encoder branch November 5, 2025 06:27
@Kay-Tian Kay-Tian mentioned this pull request Nov 5, 2025
Closed
zWaNg3 added a commit to fangyuchu/vllm that referenced this pull request Nov 7, 2025
@zWaNg3 @fangyuchu
@a798347923 @205150940 @n0gu-furiosa @chaunceyjiang @tdoublep @Flechman @benchislett @jikunshang @efimki @jeejeelee @zhang-prog @ywang96 @Isotr0py @gnovack @pwschuurman @ahao-anyscale @LucasWilkinson @aurickq @sfc-gh-aqiao @sducouedic @Lucaskabela @MatthewBonanni @andyxning @ILikeIneine @tlrmchlsmth @QiliangCui @vllmellm @li2haipeng @zhenwei-intel @markmc @yewentao256 @xiangze-arm @zhewenl @CSWYF3634076 @varun-sundar-rabindranath @yugong333 @jerryzh168 @zhuohan123 @tomeras91 @bnellnm @njhill @LyrisZhong @mgoin @vadiklyutiy @yt0428 @hmellor @ganyi1996ppo @nadavkluger @huachenheli @maleksan85 @toulzx @wangxiyuan @alex-jw-brooks @pisceskkk @KuntaiDu @minatoaquaMK2 @amirkl94 @BoyuanFeng @frost-intel @MoyanZitto @heheda12345 @ilmarkov @SageMoore @Alexei-V-Ivanov-AMD @wuhuikx @jjzhang @ZJY0516 @walterbm @gmagogsfm @sammshen @ApostaC @PaulZhang12 @noooop @windsonsea @R3hankhan123 @zou3519 @atalhens @dayeol
test_sqush (#50)
ff55dd2 
* add fault_report_addr in FaultToleranceConfig

* add handle fault&get_fault_info api

Signed-off-by: w00689259 <[email protected]>

* remove fault_report_address in CoreEngineActorManager __init__

Signed-off-by: a798347923 <[email protected]>

* ruff format

Signed-off-by: a798347923 <[email protected]>

* add handle fault&get_fault_info api

Signed-off-by: w00689259 <[email protected]>

* fix one bug.

Signed-off-by: fangyuchu <[email protected]>

* add fault_report_port in FaultToleranceConfig

Signed-off-by: a798347923 <[email protected]>

* add zmq_addr concatenate with fault_report_addr and fault_report_port

Signed-off-by: a798347923 <[email protected]>

* fault reporter bug fix

Signed-off-by: w00689259 <[email protected]>

* fault reporter bug fix

Signed-off-by: w00689259 <[email protected]>

* fault reporter bug fix

Signed-off-by: w00689259 <[email protected]>

* fault reporter bug fix

Signed-off-by: w00689259 <[email protected]>

* fault reporter bug fix

Signed-off-by: w00689259 <[email protected]>

* fault reporter bug fix

Signed-off-by: w00689259 <[email protected]>

* fix some bug

* fault reporter bug fix

Signed-off-by: w00689259 <[email protected]>

* fault reporter bug fix

Signed-off-by: w00689259 <[email protected]>

* remove fault_report_addr in FaultToleranceConfig

Signed-off-by: a798347923 <[email protected]>

* refactor: relocate method serialization functions to serial_util.py

Signed-off-by: fangyuchu <[email protected]>

* fix actor bug

* fix actor bug

* add engine_core_cmd_addr in FaultToleranceConfig

Signed-off-by: a798347923 <[email protected]>

* add and use _stop_worker_execution in EngineCoreGuard

Signed-off-by: a798347923 <[email protected]>

* add and use run in WorkerGuard

Signed-off-by: a798347923 <[email protected]>

* fix actor bug

* fix bug

* fix sentinel

* fix bug vllm/v1/engine/core.py:847: error: Missing positional argument "tp_size" in call to "EngineCoreGuard"

Signed-off-by: a798347923 <[email protected]>

* fix bug error: Missing positional arguments "length", "byteorder" in call to "to_bytes" of "int"

Signed-off-by: a798347923 <[email protected]>

* fix bug in fault tolerance mode

Signed-off-by: w00689259 <[email protected]>

* fix bug in fault tolerance mode

Signed-off-by: w00689259 <[email protected]>

* change fault_report_port to internal_fault_report_port
add external_fault_notify_port

Signed-off-by: a798347923 <[email protected]>

* change fault_report_port to internal_fault_report_port
add external_fault_notify_port

Signed-off-by: a798347923 <[email protected]>

* add _recv_cmd func
use deserialize_method_call and run_method in run func

Signed-off-by: a798347923 <[email protected]>

* Update core.py

fix bug error: Need type annotation for "kwargs" (hint: "kwargs: dict[<type>, <type>] = ...")

Signed-off-by: a798347923 <[email protected]>

* add self.ctx.term() in shutdown()

Signed-off-by: a798347923 <[email protected]>

* changed import deserialize_method_call,serialize_method_call

Signed-off-by: a798347923 <[email protected]>

* changed init worker_guard in init_device

Signed-off-by: a798347923 <[email protected]>

* Update core.py

add import serialize_method_call

Signed-off-by: a798347923 <[email protected]>

* Update gpu_worker.py

changed init WorkerGuard in init_device

Signed-off-by: a798347923 <[email protected]>

* Update gpu_worker.py

FIX BUG self.worker_guard: WorkerGuard|None = None

Signed-off-by: a798347923 <[email protected]>

* Update gpu_worker.py

fix bug error: Argument 1 to "deserialize_method_call" has incompatible type "str | None"; expected "str"  [arg-type]

Signed-off-by: a798347923 <[email protected]>

* Update gpu_worker.py

ruff format

Signed-off-by: a798347923 <[email protected]>

* Update core.py

ruff-format

Signed-off-by: a798347923 <[email protected]>

* actively send exception information

Signed-off-by: w00689259 <[email protected]>

* actively send exception information

Signed-off-by: w00689259 <[email protected]>

* actively send exception information

Signed-off-by: w00689259 <[email protected]>

* change engine_core_cmd_addr(str) to engine_core_cmd_addrs(list[str]) in EngineZmqAddresses

Signed-off-by: a798347923 <[email protected]>

* change engine_core_cmd_addr(str) to engine_core_cmd_addrs(list[str]) in EngineZmqAddresses

Signed-off-by: a798347923 <[email protected]>

* Update utils.py

delete engine_core_cmd_addr in EngineZmqAddresses

Signed-off-by: a798347923 <[email protected]>

* Remove redundant configuration: fault-pub-port

Signed-off-by: fangyuchu <[email protected]>

* Send pause instructions after receiving fault info in ClientGuard

Signed-off-by: fangyuchu <[email protected]>

* change engine_core_guard_identities from dict[int, bytes] to list[bytes]

Signed-off-by: a798347923 <[email protected]>

* fix bug "only the worker guard of engine core 0 can receive messages sent from engine core guard

Signed-off-by: a798347923 <[email protected]>

* change local_rank to rank_in_group in WorkerGuard

Signed-off-by: a798347923 <[email protected]>

* changed del self.client_cmd_registry[int(unhealthy_engine.engine_id)]

Signed-off-by: a798347923 <[email protected]>

* add gloo communication timeout

* fix some bug

* add  stateless_process_group gloo_comm_timeout

* reconstruct fault receiver&fault handler

Signed-off-by: w00689259 <[email protected]>

* fix some bug

* reconstruct fault receiver&fault handler

Signed-off-by: w00689259 <[email protected]>

* reconstruct fault receiver&fault handler

Signed-off-by: w00689259 <[email protected]>

* fix return format

Signed-off-by: w00689259 <[email protected]>

* fix return format

Signed-off-by: w00689259 <[email protected]>

* fix return format

Signed-off-by: w00689259 <[email protected]>

* add abort request

* fix some bug

* fix some bug

* fix some bug

* add dt for client guard

Signed-off-by: w00689259 <[email protected]>

* add dt for client guard

Signed-off-by: w00689259 <[email protected]>

* add dt for client guard

Signed-off-by: w00689259 <[email protected]>

* Implementation of two types of pause: a soft one by using flag signals and a hard one by aborting nccl communicators.

Signed-off-by: fangyuchu <[email protected]>

* Refine certain log forms and fix a minor bug in pause function.

Signed-off-by: fangyuchu <[email protected]>

* Refactor and abstract the recv_msg logic in CG,ECG,WG.

Signed-off-by: fangyuchu <[email protected]>

* [Frontend] Align finish_reason when tool is called with OpenAI (vllm-project#25054)

Signed-off-by: Sungyoon Jeong <[email protected]>
Co-authored-by: Chauncey <[email protected]>

* [Hybrid] Pass kernel block size to builders (vllm-project#27753)

Signed-off-by: Thomas Parnell <[email protected]>

* [Bugfix] Padded Eagle Specdec with Chunked Prefill (vllm-project#26263)

Signed-off-by: Rémi Delacourt <[email protected]>
Signed-off-by: Rémi Delacourt <[email protected]>
Signed-off-by: remi <[email protected]>
Co-authored-by: Benjamin Chislett <[email protected]>

* [XPU]Refine Dockerfile.xpu, avoid oneccl dependency issue (vllm-project#27964)

Signed-off-by: Kunshang Ji <[email protected]>

* Add and check method uuid when sending commands and receiving results.

Signed-off-by: fangyuchu <[email protected]>

* Add ORCA endpoint load metrics support (vllm-project#24905)

Signed-off-by: Misha Efimov <[email protected]>

* [CI/Build] Remove the flaky gpt-oss lora test (vllm-project#27966)

Signed-off-by: Jee Jee Li <[email protected]>

* Abstract the logic of sending instructions and waiting responses from FaultHandler

Signed-off-by: fangyuchu <[email protected]>

* [Model] Add PaddleOCR-VL Model Support  (vllm-project#27758)

Signed-off-by: zhangyue <[email protected]>
Signed-off-by: Roger Wang <[email protected]>
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: zhangyue66 <[email protected]>
Co-authored-by: Roger Wang <[email protected]>
Co-authored-by: Isotr0py <[email protected]>

* Add options in EngineCoreGuard to recv execution results from WorkerGuard

Signed-off-by: fangyuchu <[email protected]>

* Early exit for MoE LoRA kernels (vllm-project#27131)

Signed-off-by: gnovack <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>

* [Bugfix] Skip gs:// model paths for speculator detection (vllm-project#27846)

Signed-off-by: Peter Schuurman <[email protected]>

* [BUG] Make 'binary' default option for saving torch compile artifacts when using standalone_compile (vllm-project#27616)

Signed-off-by: ahao-anyscale <[email protected]>

* [CI/Testing] Add basic single node dual batch overlap test (vllm-project#27235)

Signed-off-by: Lucas Wilkinson <[email protected]>

* [Spec Decode] Integrate Suffix Decoding from Arctic Inference (vllm-project#25784)

Co-authored-by: Aurick Qiao <[email protected]>

* [Feature][Benchmarks] Support `inf` burstiness (vllm-project#26941)

Signed-off-by: Sophie du Couédic <[email protected]>

* [Bugfix][Qwen][Multimodal] Move Qwen2_5_vl sdpa to custom op and reenable compile (vllm-project#27764)

Signed-off-by: Lucas Kabela <[email protected]>

* [Bugfix] change FlashMLA reorder_batch_threshold (vllm-project#27777)

Signed-off-by: Matthew Bonanni <[email protected]>

* [Docs] add runai_streamer_sharded to LoadConfig (vllm-project#27937)

Signed-off-by: Andy Xie <[email protected]>

* Add TP parameter to attention tests (vllm-project#27683)

Signed-off-by: Matthew Bonanni <[email protected]>

* [Bugfix][plugin] fla crash on plugin (vllm-project#27322)

* [Bugfix] Fix MoE Routing Simulation (vllm-project#28002)

Signed-off-by: Tyler Michael Smith <[email protected]>

* Remove the tpu docker image nightly build. (vllm-project#27997)

Signed-off-by: Qiliang Cui <[email protected]>

* [Bugfix][ROCm] Fix ViT rotary embeddings for torch.compile compatibility on ROCm (vllm-project#27748)

Signed-off-by: vllmellm <[email protected]>

* [LoRA] Lora shrink swizzle (vllm-project#27694)

Signed-off-by: li2haipeng <[email protected]>
Signed-off-by: Haipeng Li <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>

* [Refactor] Lazy import tool_parser (vllm-project#27974)

Signed-off-by: chaunceyjiang <[email protected]>

* [NIXL][XPU] Pin NIXL version to 0.7.0 (vllm-project#27849)

Signed-off-by: zhenwei-intel <[email protected]>

* [Metrics] Enable sleep state metric outside of dev mode (vllm-project#27867)

Signed-off-by: Mark McLoughlin <[email protected]>

* [Bug] Batch invariant: Fix flash attn MLA `RuntimeError: scheduler_metadata must have shape (metadata_size)` (vllm-project#27884)

* [CPU]Improve dynamic 4bit moe performance (vllm-project#27240)

Signed-off-by: Zhang Xiangze <[email protected]>

* [CI/Build] Update LM Eval Version in AMD CI (vllm-project#27944)

Signed-off-by: zhewenli <[email protected]>

* [KV Connector] Make KVCacheConfig an explicit constructor argument (vllm-project#27887)

Signed-off-by: Mark McLoughlin <[email protected]>

* [Model] fix ernie45 reasoning_parser (vllm-project#27973)

Signed-off-by: wangyafeng <[email protected]>

* [CI/Build] Fix OpenAI API correctness on AMD CI (vllm-project#28022)

Signed-off-by: zhewenli <[email protected]>

* [BugFix][Performance] Restore flashinfer autotuning for all scenarios (vllm-project#27904)

* Support worker reinitialization after hard pause; add task queue in FaultHandler to ensure sequential task execution

Signed-off-by: fangyuchu <[email protected]>

* resolve conflicts

Signed-off-by: w00689259 <[email protected]>

* resolve conflicts

Signed-off-by: w00689259 <[email protected]>

* resolve conflicts

Signed-off-by: w00689259 <[email protected]>

* Load tuned fused_moe_lora shrink and expand kernel configs separately (vllm-project#27435)

Signed-off-by: Yu Gong <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>

* resolve conflicts

Signed-off-by: w00689259 <[email protected]>

* resolve conflicts

Signed-off-by: w00689259 <[email protected]>

* resolve conflicts

Signed-off-by: w00689259 <[email protected]>

* Support using Int4PreshuffledTensor after loading (vllm-project#26066)

Signed-off-by: Jerry Zhang <[email protected]>

* [Core] Enable StatLogger in LLMEngine (vllm-project#28020)

Signed-off-by: Zhuohan Li <[email protected]>

* [Model][Bugfix] fix pipeline parallelism support for NemotronH (vllm-project#27968)

Signed-off-by: Tomer Asida <[email protected]>

* [Model] add optimal triton fused moe configs for NemotronH MoE (vllm-project#27967)

Signed-off-by: Tomer Asida <[email protected]>

* [Kernels] Isolate modular kernel code from FusedMoEMethodBase subclasses. (vllm-project#27123)

* [BugFix] Fix incorrect preallocated sampled_token_ids tensor size (vllm-project#28025)

Signed-off-by: Nick Hill <[email protected]>

* [Perf] SM100 - add swap AB optimization to CUTLASS FP8 GEMM (vllm-project#27284)

Signed-off-by: Faqin Zhong <[email protected]>
Co-authored-by: Faqin Zhong <[email protected]>
Co-authored-by: Michael Goin <[email protected]>

* [PERF] Decouple projections from GDN custom op (vllm-project#27512)

Signed-off-by: Vadim Gimpelson <[email protected]>

* [model] Add support for openPangu_Ultra_MoE (vllm-project#27521)

Signed-off-by: yuantao <[email protected]>
Signed-off-by: yt0428 <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>

* [PerfFix] Avoid separate thread for MP executor shm spin (vllm-project#28012)

Signed-off-by: Nick Hill <[email protected]>

* [AsyncScheduling] Don't schedule past request max_tokens (vllm-project#27922)

Signed-off-by: Nick Hill <[email protected]>

* Remove deprecated `--rope-scaling` and `--rope-theta` (vllm-project#28006)

Signed-off-by: Harry Mellor <[email protected]>

* [ROCm][Perf] New design on ROCm AITER MHA backend Implementation (vllm-project#25763)

Signed-off-by: ganyi <[email protected]>

* Added disable rule to track files under benchmarks/lib (vllm-project#28048)

Signed-off-by: Nadav Kluger <[email protected]>

* [Multimodal] Make MediaConnector extensible. (vllm-project#27759)

Signed-off-by: Chenheli Hua <[email protected]>

* [ROCm] gemm_a16w16 upstreaming (vllm-project#26969)

Signed-off-by: Aleksandr Malyshev <[email protected]>
Co-authored-by: Aleksandr Malyshev <[email protected]>

* Revert "[PERF] Decouple projections from GDN custom op" (vllm-project#28080)

Signed-off-by: Vadim Gimpelson <[email protected]>

* add engine core ut

Signed-off-by: w00689259 <[email protected]>

* add engine core ut

Signed-off-by: w00689259 <[email protected]>

* [Qwen3-Next] MOE configs for A100-SXM4-80GB TP4 TP8 (vllm-project#27740)

* [XPU] Add gpt-oss model support for Intel GPU (vllm-project#27786)

Signed-off-by: Kunshang Ji <[email protected]>

* [CI/Build] Enable some fixed tests in AMD CI (vllm-project#28078)

Signed-off-by: zhewenli <[email protected]>

* [V0 deprecation] Remove VLLM_USE_V1 usage in most modules (vllm-project#27955)

Signed-off-by: wangxiyuan <[email protected]>

* [Bugfix] Fix encoder-only model support for transformers backend (vllm-project#28021)

Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Harry Mellor <[email protected]>
Co-authored-by: Harry Mellor <[email protected]>

* [BugFix] Fix DCP Assert (AssertionError: DCP not support reorder_batch_threshold > 1 now.) (vllm-project#28100)

Signed-off-by: Lucas Wilkinson <[email protected]>

* [Model, Core] Support Granite Speech & LoRA for STT (vllm-project#24455)

* [Refactor] Lazy-loaded reasoning_parser (vllm-project#28092)

Signed-off-by: chaunceyjiang <[email protected]>

* [Refactor] to simplify and extract the shared logic between chat completion and responses (vllm-project#27961)

Signed-off-by: chaunceyjiang <[email protected]>

* [bugfix] fix wrong `dcp_local_seq_lens` calc (vllm-project#27518)

Signed-off-by: Qiu <[email protected]>

* [Hybrid allocator + kv connector] revert connector test changes related to hybrid allocator (vllm-project#28011)

Signed-off-by: KuntaiDu <[email protected]>

* [Misc] fix import error for DeepSeekR1ReasoningParser (vllm-project#28114)

Signed-off-by: chaunceyjiang <[email protected]>

* Fix excessive logging noise by reducing the log level of the MinimaxM2ToolParser import success message (vllm-project#27635)

Signed-off-by: minatoaquaMK2 <[email protected]>

* Bugfix: Cutlass FP8 FusedMoE bad scaling factors (vllm-project#27255)

Signed-off-by: Amir Klein <[email protected]>
Co-authored-by: Tyler Michael Smith <[email protected]>
Co-authored-by: Michael Goin <[email protected]>

* [Graph Partition][Cache] Use inductor partition ops config (vllm-project#27702)

Signed-off-by: Boyuan Feng <[email protected]>

* [XPU] Enable custom routing functions in IPEX for Llama4 (vllm-project#28004)

Signed-off-by: frost-intel <[email protected]>

* add kimi reasoning parser (vllm-project#28128)

Signed-off-by: wangzhengtao <[email protected]>
Co-authored-by: wangzhengtao <[email protected]>

* [DCP] check return_lse for all layers in dcp (vllm-project#27929)

Signed-off-by: Chen Zhang <[email protected]>

* [BugFix] Support EP/DP + EPLB with MTP (vllm-project#25311)

Signed-off-by: ilmarkov <[email protected]>
Signed-off-by: Sage Moore <[email protected]>
Co-authored-by: Sage Moore <[email protected]>
Co-authored-by: Tyler Michael Smith <[email protected]>
Co-authored-by: Lucas Wilkinson <[email protected]>

* Enabling cooperative multi-gpu tests on multi-gpu nodes (vllm-project#27986)

Signed-off-by: Alexei V. Ivanov <[email protected]>

* [ROCm][MLA] Support block-size > 1 for AITER MLA backend  (vllm-project#27224)

Signed-off-by: ganyi <[email protected]>
Co-authored-by: wuhuikx <[email protected]>

* [Bugfix] Validate custom logits processor xargs for online serving (vllm-project#27560)

Signed-off-by: Isotr0py <[email protected]>

* [misc] add vLLM Beijing Meetup (vllm-project#28127)

Signed-off-by: Jiaju Zhang <[email protected]>

* [Kernel] Fuse computation of g and beta for Gated Delta Net (vllm-project#28095)

Signed-off-by: zjy0516 <[email protected]>

* [Core] add support for reasoning parser plugins (vllm-project#28075)

Signed-off-by: walter beller-morales <[email protected]>

* [Bugfix] vLLM should check Inductor config for compile cache enablement status (vllm-project#27637)

Signed-off-by: Yanan Cao <[email protected]>

* [FlashInfer] Avoid FlashInfer block_size 16 + head_size 256 on blackwell (vllm-project#27994)

Signed-off-by: Chen Zhang <[email protected]>

* [CI]: Add LMCacheConnector Unit Tests (vllm-project#27852)

Signed-off-by: Samuel Shen <[email protected]>
Co-authored-by: Samuel Shen <[email protected]>
Co-authored-by: Yihua Cheng <[email protected]>

* [Feature] Extend batch invariant torch.compile to B200 (vllm-project#27856)

Signed-off-by: PaulZhang12 <[email protected]>

* [Bugfix] Fix Qwen3-Reranker-8B load (vllm-project#28117)

Signed-off-by: wang.yuqi <[email protected]>

* [Docs] Clean up README_TUNING.md (vllm-project#28088)

Signed-off-by: windsonsea <[email protected]>

* [Hardware][IBM Z] Optimize s390x Dockerfile (vllm-project#28023)

Signed-off-by: Rehan Khan <[email protected]>

* [Chore] Remove Nemotron-Nano-VL config copy (vllm-project#28126)

Signed-off-by: Isotr0py <[email protected]>

* [Docs] Add guide to debugging vLLM-torch.compile integration (vllm-project#28094)

Signed-off-by: Richard Zou <[email protected]>

* [Feature]: Add corrupted request metric to V1 metrics system. (vllm-project#27306)

Signed-off-by: atalhens <[email protected]>

* [CI/Build] Update checking logic in cutlass_group_gemm_supported  (vllm-project#27948)

Signed-off-by: zhewenli <[email protected]>

* [CI/Build] Fix `test_defaults_with_usage_context` in AMD CI (vllm-project#27926)

Signed-off-by: zhewenli <[email protected]>

* [Core][Hybrid allocator + connector 2/n] Unify `remove_skipped_blocks` by `get_last_useful_token` (vllm-project#25431)

Signed-off-by: KuntaiDu <[email protected]>

* [Debugging] Add annotation for easier trace analysis (vllm-project#22496)

* [PERF] Decouple projections from GDN custom op. Attempt 2 (vllm-project#28083)

Signed-off-by: Vadim Gimpelson <[email protected]>

* [Bug] Fix cpu disable shared_experts `VLLM_DISABLE_SHARED_EXPERTS_STREAM` (vllm-project#28157)

Signed-off-by: yewentao256 <[email protected]>

* [Bug] Fix env string `"0"` same to `True` (vllm-project#28159)

Signed-off-by: yewentao256 <[email protected]>

* Ensure WorkerGuard command execution returns result; fix missing set_device when TP>1

Signed-off-by: fangyuchu <[email protected]>

* [Feature] Enable TP + EP `shared_experts` overlap with router, 3.7% E2E performance improvement (vllm-project#28164)

Signed-off-by: yewentao256 <[email protected]>

* [CI Failure] `nm-testing/Qwen2-0.5B-Instruct-FP8-SkipQKV` was removed from HF. Skip it in tests (vllm-project#28170)

Signed-off-by: Vadim Gimpelson <[email protected]>

* [Misc] Remove the duplicate code (vllm-project#28111)

Signed-off-by: chaunceyjiang <[email protected]>

* rename& format logger

Signed-off-by: w00689259 <[email protected]>

* rename& format logger

Signed-off-by: w00689259 <[email protected]>

* feat(nccl): enable non-blocking NCCL communicators to support ncclCommAbort

Signed-off-by: fangyuchu <[email protected]>

---------

Signed-off-by: w00689259 <[email protected]>
Signed-off-by: a798347923 <[email protected]>
Signed-off-by: fangyuchu <[email protected]>
Signed-off-by: zWaNg3 <[email protected]>
Signed-off-by: a798347923 <[email protected]>
Signed-off-by: Sungyoon Jeong <[email protected]>
Signed-off-by: Thomas Parnell <[email protected]>
Signed-off-by: Rémi Delacourt <[email protected]>
Signed-off-by: Rémi Delacourt <[email protected]>
Signed-off-by: remi <[email protected]>
Signed-off-by: Kunshang Ji <[email protected]>
Signed-off-by: Misha Efimov <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: zhangyue <[email protected]>
Signed-off-by: Roger Wang <[email protected]>
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: zhangyue66 <[email protected]>
Signed-off-by: gnovack <[email protected]>
Signed-off-by: Peter Schuurman <[email protected]>
Signed-off-by: ahao-anyscale <[email protected]>
Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: Sophie du Couédic <[email protected]>
Signed-off-by: Lucas Kabela <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
Signed-off-by: Andy Xie <[email protected]>
Signed-off-by: Tyler Michael Smith <[email protected]>
Signed-off-by: Qiliang Cui <[email protected]>
Signed-off-by: vllmellm <[email protected]>
Signed-off-by: li2haipeng <[email protected]>
Signed-off-by: Haipeng Li <[email protected]>
Signed-off-by: chaunceyjiang <[email protected]>
Signed-off-by: zhenwei-intel <[email protected]>
Signed-off-by: Mark McLoughlin <[email protected]>
Signed-off-by: Zhang Xiangze <[email protected]>
Signed-off-by: zhewenli <[email protected]>
Signed-off-by: wangyafeng <[email protected]>
Signed-off-by: Yu Gong <[email protected]>
Signed-off-by: Jerry Zhang <[email protected]>
Signed-off-by: Zhuohan Li <[email protected]>
Signed-off-by: Tomer Asida <[email protected]>
Signed-off-by: Nick Hill <[email protected]>
Signed-off-by: Faqin Zhong <[email protected]>
Signed-off-by: Vadim Gimpelson <[email protected]>
Signed-off-by: yuantao <[email protected]>
Signed-off-by: yt0428 <[email protected]>
Signed-off-by: Harry Mellor <[email protected]>
Signed-off-by: ganyi <[email protected]>
Signed-off-by: Nadav Kluger <[email protected]>
Signed-off-by: Chenheli Hua <[email protected]>
Signed-off-by: Aleksandr Malyshev <[email protected]>
Signed-off-by: wangxiyuan <[email protected]>
Signed-off-by: Qiu <[email protected]>
Signed-off-by: KuntaiDu <[email protected]>
Signed-off-by: minatoaquaMK2 <[email protected]>
Signed-off-by: Amir Klein <[email protected]>
Signed-off-by: Boyuan Feng <[email protected]>
Signed-off-by: frost-intel <[email protected]>
Signed-off-by: wangzhengtao <[email protected]>
Signed-off-by: Chen Zhang <[email protected]>
Signed-off-by: ilmarkov <[email protected]>
Signed-off-by: Sage Moore <[email protected]>
Signed-off-by: Alexei V. Ivanov <[email protected]>
Signed-off-by: Jiaju Zhang <[email protected]>
Signed-off-by: zjy0516 <[email protected]>
Signed-off-by: walter beller-morales <[email protected]>
Signed-off-by: Yanan Cao <[email protected]>
Signed-off-by: Samuel Shen <[email protected]>
Signed-off-by: PaulZhang12 <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Signed-off-by: windsonsea <[email protected]>
Signed-off-by: Rehan Khan <[email protected]>
Signed-off-by: Richard Zou <[email protected]>
Signed-off-by: atalhens <[email protected]>
Signed-off-by: yewentao256 <[email protected]>
Co-authored-by: fangyuchu <[email protected]>
Co-authored-by: a798347923 <[email protected]>
Co-authored-by: w00689259 <[email protected]>
Co-authored-by: fangyuchu <[email protected]>
Co-authored-by: TianZhuo <[email protected]>
Co-authored-by: a798347923 <[email protected]>
Co-authored-by: Sungyoon Jeong <[email protected]>
Co-authored-by: Chauncey <[email protected]>
Co-authored-by: Thomas Parnell <[email protected]>
Co-authored-by: Rémi Delacourt <[email protected]>
Co-authored-by: Benjamin Chislett <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>
Co-authored-by: Misha Efimov <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>
Co-authored-by: zhang-prog <[email protected]>
Co-authored-by: Roger Wang <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
Co-authored-by: gnovack <[email protected]>
Co-authored-by: pwschuurman <[email protected]>
Co-authored-by: ahao-anyscale <[email protected]>
Co-authored-by: Lucas Wilkinson <[email protected]>
Co-authored-by: Aurick Qiao <[email protected]>
Co-authored-by: Aurick Qiao <[email protected]>
Co-authored-by: Sophie du Couédic <[email protected]>
Co-authored-by: Lucas Kabela <[email protected]>
Co-authored-by: Matthew Bonanni <[email protected]>
Co-authored-by: Ning Xie <[email protected]>
Co-authored-by: Hank_ <[email protected]>
Co-authored-by: Tyler Michael Smith <[email protected]>
Co-authored-by: QiliangCui <[email protected]>
Co-authored-by: vllmellm <[email protected]>
Co-authored-by: li2haipeng <[email protected]>
Co-authored-by: liuzhenwei <[email protected]>
Co-authored-by: Mark McLoughlin <[email protected]>
Co-authored-by: Wentao Ye <[email protected]>
Co-authored-by: xiangze-arm <[email protected]>
Co-authored-by: Zhewen Li <[email protected]>
Co-authored-by: CSWYF3634076 <[email protected]>
Co-authored-by: Varun Sundar Rabindranath <[email protected]>
Co-authored-by: yugong333 <[email protected]>
Co-authored-by: Jerry Zhang <[email protected]>
Co-authored-by: Zhuohan Li <[email protected]>
Co-authored-by: tomeras91 <[email protected]>
Co-authored-by: bnellnm <[email protected]>
Co-authored-by: Nick Hill <[email protected]>
Co-authored-by: lyrisz <[email protected]>
Co-authored-by: Faqin Zhong <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Vadim Gimpelson <[email protected]>
Co-authored-by: yt0428 <[email protected]>
Co-authored-by: Harry Mellor <[email protected]>
Co-authored-by: Pleaplusone <[email protected]>
Co-authored-by: nadavkluger <[email protected]>
Co-authored-by: Chenheli Hua <[email protected]>
Co-authored-by: Aleksandr Malyshev <[email protected]>
Co-authored-by: Aleksandr Malyshev <[email protected]>
Co-authored-by: tou <[email protected]>
Co-authored-by: wangxiyuan <[email protected]>
Co-authored-by: Alex Brooks <[email protected]>
Co-authored-by: Qiu <[email protected]>
Co-authored-by: Kuntai Du <[email protected]>
Co-authored-by: Eric Yue <[email protected]>
Co-authored-by: amirkl94 <[email protected]>
Co-authored-by: Boyuan Feng <[email protected]>
Co-authored-by: Frost Mitchell <[email protected]>
Co-authored-by: bigmoyan <[email protected]>
Co-authored-by: wangzhengtao <[email protected]>
Co-authored-by: Chen Zhang <[email protected]>
Co-authored-by: Ilya Markov <[email protected]>
Co-authored-by: Sage Moore <[email protected]>
Co-authored-by: Alexei-V-Ivanov-AMD <[email protected]>
Co-authored-by: wuhuikx <[email protected]>
Co-authored-by: Jiaju Zhang <[email protected]>
Co-authored-by: Jiangyun Zhu <[email protected]>
Co-authored-by: Walter Beller-Morales <[email protected]>
Co-authored-by: gmagogsfm <[email protected]>
Co-authored-by: Samuel Shen <[email protected]>
Co-authored-by: Samuel Shen <[email protected]>
Co-authored-by: Yihua Cheng <[email protected]>
Co-authored-by: Paul Zhang <[email protected]>
Co-authored-by: wang.yuqi <[email protected]>
Co-authored-by: Michael Yao <[email protected]>
Co-authored-by: R3hankhan <[email protected]>
Co-authored-by: Richard Zou <[email protected]>
Co-authored-by: Snehlata <[email protected]>
Co-authored-by: Dayeol Lee <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@NickLucche NickLucche NickLucche left review comments

@heheda12345 heheda12345 heheda12345 approved these changes

@DarkLight1337 DarkLight1337 DarkLight1337 left review comments

@hmellor hmellor hmellor approved these changes

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

Transformers backend
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Isotr0py @NickLucche @hmellor @heheda12345 @DarkLight1337 @vllm-bot