[Feat] Router-side queuing support

[allytotheson]

Summary

This PR introduces an asynchronous EndpointQueueManager to handle request queueing and dispatch for LLM inference endpoints. It enables queue-based request routing to improve Quality of Experience (QoE), particularly under high load, while supporting immediate dispatch when endpoints are free.

Key Features

• Per-endpoint request queues with PriorityQueue and asyncio.Condition
• Scheduler loop per endpoint that handles dispatch based on availability
• Optional queue toggling via a configuration flag
• Rerouting of stale requests that exceed a configurable wait threshold
• Scaffolding in place for future session aware and/or KV cache integration when handling stale requests
• Graceful shutdown of endpoint schedulers

Highlights

• Requests are either:
  • Enqueued if the endpoint is overloaded
  • Immediately dispatched if the endpoint is free or queueing is disabled
• Rerouting logic kicks in for requests that exceed max_queue_wait_time

Future Work

• Implement KV cache-aware, session-based stale-request rerouting
• Knapsack-style group dispatch logic for optimized batch handling

[gemini-code-assist]

Summary of Changes

Hello @allytotheson, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

I've introduced a significant enhancement to the router by adding comprehensive router-side queuing capabilities for LLM inference requests. This change is designed to drastically improve the Quality of Experience (QoE) for users, especially under heavy load conditions. Instead of immediately rejecting or failing requests when an endpoint is busy, the system will now intelligently queue them, dispatching them as soon as resources become available. This also includes logic to reroute requests that have been waiting too long, ensuring better overall throughput and responsiveness without requiring manual intervention.

Highlights

• Per-Endpoint Request Queues: I've implemented an EndpointQueueManager that maintains a PriorityQueue and an asyncio.Condition for each LLM inference endpoint. This allows requests to be intelligently queued and dispatched.
• Asynchronous Scheduler Loops: A dedicated asynchronous scheduler loop runs for each endpoint, responsible for dispatching requests from its queue when the endpoint becomes available, based on configurable load thresholds (e.g., number of running requests, GPU usage).
• Stale Request Rerouting: Requests that remain in the queue beyond a max_queue_wait_time threshold are automatically rerouted to a different available endpoint. This logic is designed to improve responsiveness and prevent requests from getting stuck.
• Configurable Queue Toggling: The queuing mechanism can be toggled on or off via a configuration flag, providing flexibility for different deployment scenarios. Even when disabled, the manager is instantiated.
• Future-Proofing for Session Affinity and KV Cache: The system includes scaffolding for future enhancements, such as session-aware rerouting and integration with KV cache, to further optimize request handling for stale requests.
• Graceful Shutdown: The queue manager ensures graceful shutdown of all its background scheduler tasks, preventing resource leaks and ensuring clean application termination.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a robust queuing mechanism for handling LLM requests, which is a great feature for improving service reliability under load. The implementation is well-structured, using asyncio features effectively. My review focuses on potential race conditions, correctness bugs in shutdown and rerouting logic, and improvements in argument parsing and maintainability. Key areas for improvement include using asyncio.Lock instead of threading.Lock, fixing a potential KeyError in request routing, and correcting the graceful shutdown procedure.

[gemini-code-assist]

The queue manager shutdown logic has two issues:

1. queue_manager.close() is an async function and must be awaited for a graceful shutdown.
2. get_queue_manager() raises a ValueError if the manager is not initialized, it does not return None. The if queue_manager is not None: check is therefore ineffective. You should use a try...except ValueError block to handle the case where the queue manager was not initialized.

    # Close the queue manager
    try:
        queue_manager = get_queue_manager()
        if queue_manager:
            logger.info("Closing per endpoint queues and tasks")
            await queue_manager.close()
    except ValueError:
        # Queue manager was not initialized.
        pass

[zerofishnoodles]

Hi, I am wondering the difference between it and vllm scheduler. There is already a request scheduler inside vllm, why do we need this?

[allytotheson]

Hi, I am wondering the difference between it and vllm scheduler. There is already a request scheduler inside vllm, why do we need this?

Hi, thanks for the question! Is the vllm scheduler you're referring to the one that manages token-level execution at the backend?

This queueing system focuses on admission control at the router level, before the requests are sent to the backend. The main goal is to avoid overwhelming any single backend. I have default thresholds like 10 concurrent requests and 95% GPU usage as signals to hold off on dispatching new requests.
What makes this useful is that it enables strategies like rerouting requests to another endpoint if they’ve been stuck in a queue too long. It also opens the door for more advanced dispatch logic — for example, priority-based queueing (even though currently everything is treated as high priority).

That said, I'm definitely open to exploring smarter strategies that don’t strictly block requests when thresholds are hit to allow it to work better with the request scheduler.

[zerofishnoodles]

I think this one is a super good feature! Thanks for contributing! Could you make the enable_queue default value false? And then I think we are good to go for now. Hoping to see the following changes.

[zerofishnoodles]

Can you set default to false since this feature is not totally finished?

[lucas-tucker]

Has this been tested and has shown to improve performance? Router-side queuing makes sense, but this appears to be more of a priority mechanism on the endpoints as route_general_request is not touched.

[zerofishnoodles]

LGTM