[CI/Build]Add eval config for Qwen3-235B-A22B-Instruct-2507-FP8

[hl475]

Purpose

• This PR adds eval config for Qwen3-235B-A22B-Instruct-2507-FP8
• We rename .buildkite/lm-eval-harness/configs/models-large-h100.txt to use .buildkite/lm-eval-harness/configs/models-large-hopper.txt per suggestion.
• Add a new test/label label: LM Eval Large Models (H100) # optional to run configs/models-large-hopper.txt

Test Plan

pytest -s -v .buildkite/lm-eval-harness/test_lm_eval_correctness.py \
   --config-list-file .buildkite/lm-eval-harness/configs/models-large-hopper.txt \
   --tp-size 4

Test Result

WIP

VLLM_USE_DEEP_GEMM=1

1 passed, 104 warnings in 3062.67s (0:51:02)
https://buildkite.com/vllm/ci/builds/36443/steps/canvas?sid=019a272c-b17c-4404-96e9-0e83356d6bb8

VLLM_USE_DEEP_GEMM=1

1 passed, 104 warnings in 3041.13s (0:50:41)
https://buildkite.com/vllm/ci/builds/36457/steps/canvas?sid=019a27b5-4ec8-4166-8bba-091034e14805

VLLM_USE_DEEP_GEMM=0

1 passed, 104 warnings in 1900.52s (0:31:40)
https://buildkite.com/vllm/ci/builds/36470/steps/canvas?sid=019a2826-8d7a-4453-925b-472e24acf179

VLLM_USE_DEEP_GEMM=0 + kv_cache_dtype=fp8

1 passed, 104 warnings in 1445.30s (0:24:05)
https://buildkite.com/vllm/ci/builds/36607/steps/canvas?sid=019a2b72-d030-4a51-aefa-ff0838ffc8fb

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[hl475]

cc @yeqcharlotte @zhewenl

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[zhewenl]

cc @yeqcharlotte @zhewenl

LGTM, have we verified by triggering an example CI on buildkite?

[hl475]

cc @yeqcharlotte @zhewenl

LGTM, have we verified by triggering an example CI on buildkite?

Thanks! I did it once over the weekend. Let me schedule a new one https://buildkite.com/vllm/ci/builds/35613/steps/canvas

LM Eval Small Models https://buildkite.com/vllm/ci/builds/35613/steps/canvas?sid=019a0362-abbd-407f-8d10-243f063b0792

LM Eval Large Models (H200) https://buildkite.com/vllm/ci/builds/35613/steps/canvas?jid=019a04d4-029e-4ed2-90e6-ccc726f215d1

[zhewenl]

Do we also want to test llama4? Or we can just start with Qwen?

[yeqcharlotte]

how long does it take with and without VLLM_USE_DEEP_GEMM? sounds like qwen model is not doing good with deepgemm? cc: @minosfuture @houseroad

[hl475]

how long does it take with and without VLLM_USE_DEEP_GEMM? sounds like qwen model is not doing good with deepgemm? cc: @minosfuture @houseroad

there seems some build issue at the moment (sample failure, error msg Failed to fetch http://security.ubuntu.com/ubuntu) - will have to retry later

[hl475]

@yeqcharlotte - updated the test result in the Test Result section.

The fastest I can get using H100 CI is about 24 minutes. As a comparison, the same setting spent about 15 minutes on my local H100. One obvious difference between the two H100 are - my local H100 is 96GB and CI H100 is 80GB.

[yeqcharlotte]

let's get it started!