[CI]: reduce HTTP calls inside entrypoints openai tests

[AzizCode92]

Purpose

Previously, tests would repeatedly make HTTP calls to HF_HUB at different test modules even when the lora model and modules are downloaded.
As this would slow down our CI pipeline and increase the risk of network failures, I propose this PR.

This refactoring introduces a session-scoped pytest fixture that downloads and caches the required models and LoRAs a single time. All tests now rely on this fixture, which eliminates redundant HTTP calls and makes the CI pipeline faster and more reliable.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[njhill]

Thanks @AzizCode92! Are you in vllm slack? We have a channel #ci-sprint where we've just started to coordinate these efforts.

[njhill]

@AzizCode92 we also want to assess whether we really need to use LoRA in all of the tests. It would be good to swap out the non-LoRA model for a tiny one if possible, see #23667

[jeejeelee]

@AzizCode92 we also want to assess whether we really need to use LoRA in all of the tests. It would be good to swap out the non-LoRA model for a tiny one if possible, see #23667

Makes sense

[AzizCode92]

Thanks @AzizCode92! Are you in vllm slack? We have a channel #ci-sprint where we've just started to coordinate these efforts.

Hi @njhill, no I just sent a request to join it.

@AzizCode92 we also want to assess whether we really need to use LoRA in all of the tests. It would be good to swap out the non-LoRA model for a tiny one if possible, see #23667

Thanks for clarifying. Just to confirm my understanding, you're suggesting we consolidate LoRA-specific tests into tests/entrypoints/openai/test_lora_adapters.py and remove them from other test files to avoid redundancy.

For instance, tests/entrypoints/openai/test_chat.py currently tests the HuggingFaceH4/zephyr-7b-beta model with LoRA modules. Since this functionality is already covered in the dedicated test_lora_adapters.py suite, I can proceed with removing that specific test case from test_chat.py + ofc replacing HuggingFaceH4/zephyr-7b-beta with a tiny model from #23456 (comment).

Does this align with what you had in mind?

[robertgshaw2-redhat]

Thanks @AzizCode92! Are you in vllm slack? We have a channel #ci-sprint where we've just started to coordinate these efforts.

Hi @njhill, no I just sent a request to join it.

@AzizCode92 we also want to assess whether we really need to use LoRA in all of the tests. It would be good to swap out the non-LoRA model for a tiny one if possible, see #23667

Thanks for clarifying. Just to confirm my understanding, you're suggesting we consolidate LoRA-specific tests into tests/entrypoints/openai/test_lora_adapters.py and remove them from other test files to avoid redundancy.

For instance, tests/entrypoints/openai/test_chat.py currently tests the HuggingFaceH4/zephyr-7b-beta model with LoRA modules. Since this functionality is already covered in the dedicated test_lora_adapters.py suite, I can proceed with removing that specific test case from test_chat.py + ofc replacing HuggingFaceH4/zephyr-7b-beta with a tiny model from #23456 (comment).

Does this align with what you had in mind?

yep --- we need to be careful that we aren't removing some coverage, but ideally there should be a single file for LoRA. Then we can have a single shared common fixture for the openai tests that that uses a tiny model. Removing LoRA should help a lot with the sharing across the other test groups

[jeejeelee]

Previously, tests would repeatedly download assets from the Hugging Face Hub, slowing down CI and increasing the risk of network failures.
This refactoring introduces a session-scoped pytest fixture that downloads and caches the required models and LoRAs a single time. All tests now rely on this fixture, which eliminates redundant downloads and makes the CI pipeline faster and more reliable.

IIUC, LoRA weight should be the same as the model - if it's already downloaded, it won't download again repeatedly, so this PR won't work. See : https://buildkite.com/vllm/ci/builds/28468#0198e9bd-43ae-42d6-8710-12986ab9b173/210

@DarkLight1337 If I am wrong, please correct me

[DarkLight1337]

I think this PR can still help by reducing the number of HTTP requests to check the updated status of each file, even though weights won't be re-downloaded

[AzizCode92]

Previously, tests would repeatedly download assets from the Hugging Face Hub, slowing down CI and increasing the risk of network failures.
This refactoring introduces a session-scoped pytest fixture that downloads and caches the required models and LoRAs a single time. All tests now rely on this fixture, which eliminates redundant downloads and makes the CI pipeline faster and more reliable.

IIUC, LoRA weight should be the same as the model - if it's already downloaded, it won't download again repeatedly, so this PR won't work. See : https://buildkite.com/vllm/ci/builds/28468#0198e9bd-43ae-42d6-8710-12986ab9b173/210

@DarkLight1337 If I am wrong, please correct me

Thanks for the feedback.
As @DarkLight1337 pointed, this PR will minimize the HTTP calls for our tests. Although its a minor improvement but I think still necessary to improve the speed of the CI pipeline.
snapshot_download() will make these HTTP calls even when the files are already cached. So I thought we get rid of this call from the modules level and make a session fixture for it to make it happen once for all tests.

[njhill]

I think this PR can still help by reducing the number of HTTP requests to check the updated status of each file, even though weights won't be re-downloaded

We are hoping to eliminate all of the http calls globally anyhow #23451

[AzizCode92]

Once the tiny lora model is ready, it will replace the zephyr model and its lora modules.
#23456 (comment)

[hmellor]

Do we still use the prompt adapter files? This feature was removed a while ago.

[AzizCode92]

Absolutelty! I will remove it

[jeejeelee]

Previously, tests would repeatedly download assets from the Hugging Face Hub, slowing down CI and increasing the risk of network failures.
This refactoring introduces a session-scoped pytest fixture that downloads and caches the required models and LoRAs a single time. All tests now rely on this fixture, which eliminates redundant downloads and makes the CI pipeline faster and more reliable.

IIUC, LoRA weight should be the same as the model - if it's already downloaded, it won't download again repeatedly, so this PR won't work. See : https://buildkite.com/vllm/ci/builds/28468#0198e9bd-43ae-42d6-8710-12986ab9b173/210
@DarkLight1337 If I am wrong, please correct me

Thanks for the feedback. As @DarkLight1337 pointed, this PR will minimize the HTTP calls for our tests. Although its a minor improvement but I think still necessary to improve the speed of the CI pipeline. snapshot_download() will make these HTTP calls even when the files are already cached. So I thought we get rid of this call from the modules level and make a session fixture for it to make it happen once for all tests.

That makes sense, but what if you set local_files_only=True?
If it's to reduce HTTP calls, please modify your PR title.

[jeejeelee]

I have removed tests/entrypoints/llm/test_generate_multiple_loras.py, please resolve the branch conflicit ,then overall LGTM

[jeejeelee]

overall LGTM

@DarkLight1337 @hmellor please take another look, thank you

[hmellor]

Can we also remove the instances of LORA_NAME in all the files where zephyr_lora_files has been removed? It doesn't appear to be used anywhere but could confuse future maintainers.