-
Notifications
You must be signed in to change notification settings - Fork 11.7k
Feature/queue pause resume #57800
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 12.x
Are you sure you want to change the base?
Feature/queue pause resume #57800
Conversation
This commit introduces a new feature that allows pausing and resuming individual queues without stopping workers, providing granular control over queue processing.
Features:
- Queue::pause($queue, $ttl) - Pause a specific queue with optional TTL
- Queue::resume($queue) - Resume a paused queue
- Queue::isPaused($queue) - Check if a queue is paused
- Queue::getPausedQueues() - Get list of all paused queues
Artisan Commands:
- queue:pause {queue} - Pause a queue
- queue:resume {queue} - Resume a queue
- queue:pause:list - List all paused queues
Implementation:
- Modified Worker to skip paused queues when looking for jobs
- Uses cache to track paused queue status with configurable TTL
- Works with all queue drivers (Redis, Database, SQS, etc.)
- No jobs are lost - they remain in queue until resumed
- Fully backwards compatible with existing queue functionality
Files Modified:
- src/Illuminate/Queue/QueueManager.php
- src/Illuminate/Queue/Worker.php
- src/Illuminate/Foundation/Providers/ArtisanServiceProvider.php
Files Created:
- src/Illuminate/Queue/Console/PauseCommand.php
- src/Illuminate/Queue/Console/ResumeCommand.php
- src/Illuminate/Queue/Console/PauseListCommand.php
- QUEUE_PAUSE_RESUME_FEATURE.md
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Update the Factory interface to include the new pause/resume methods. This resolves IDE warnings and provides proper type hints for the queue pause/resume functionality. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…de methods for queue management
|
I have had something like this for years on a project to skip jobs after reaching an API rate limit to avoid attempting several jobs in a row. It also allows having a worker working on more than one queue while the other one is paused/blocked (in my implementation I call it blocked). Locally I have a custom queue driver that extends from the And a job middleware to block/pause the queue when a 429 response is caught. It will be great to have this in the framework. Also, it is a much more generic solution than mine. Thanks a lot! |
| { | ||
| $cache = $this->app['cache']->store(); | ||
|
|
||
| $cache->put("queue_paused:{$queue}", true, $ttl); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Operation is not atomic here. In some concurrent situation (pausing two queues same time) threir may be such case:
- first process getPausedQueues returns []
- second proccess getPausedQueues returns []
- first process storePausedQueuesList sets ['queue1']
- second process storePausedQueuesList sets ['queue2']
finaly queue_paused_list has ['queue2'] and not ['queue1', 'queue2'] as expected.
One way is to use some sort of locks or isolation here.
But maybe it would be better for siplicicty not to provide PauseList functionality at all. Pausing and resuming certain queue (and single isolated cache key) would be enough. Currently framwork does not have any functions to get all existing queues list, so listing just paused queues is redundant.
| * @param int $ttl | ||
| * @return void | ||
| */ | ||
| public function pause($queue, $ttl = 86400) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very strange solution to pause just for one day. Why not week, year, or 30 seconds? It's better to pass null as default, meaning by default pausing queue until it's resumed manualy.
For other cases developers still will have a way to pass any ttl they want in certain case
src/Illuminate/Queue/Worker.php
Outdated
|
|
||
| foreach (explode(',', $queue) as $index => $queue) { | ||
| // Skip paused queues | ||
| if ($this->isQueuePaused($queue)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider situations, where you have three default queue on three different connections. In current solution, trying to pause it, you will pause all three queues, leaving no way to pause just one.
You should take connection name into account too, to point exactly to specific queue.
vadimonus
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting idea, but solution has flaws
|
I think I would want the ability to pause on both connection and queue with syntax similar to: https://laravel.com/docs/12.x/queues#monitoring-your-queues
|
… pause list and atomic updates.
Hi Taylor, Thank you for the feedback! I've updated the implementation to support the Changes made: ✅ Commands now accept Examples: php artisan queue:pause redis:default
php artisan queue:pause database:emails --ttl=3600
php artisan queue:resume redis:defaultThe implementation uses a |
Hi vadimonus, Thank you for the detailed code review! I've addressed all three issues you identified: 1. Race Condition ✅You were absolutely right about the non-atomic operations. I've implemented proper locking using $lock = $cache->lock('queue_paused_list_lock', 10);
try {
$lock->block(5);
$pausedQueues = $this->getPausedQueues();
// ... modify list atomically ...
$this->storePausedQueuesList($pausedQueues);
} finally {
$lock->release();
}This ensures atomic read-modify-write operations and prevents the concurrent modification issue you described. 2. TTL Default Value ✅Changed from 1 day to public function pause($connection, $queue, $ttl = null)
{
if ($ttl === null) {
$cache->forever("queue_paused:{$connection}:{$queue}", true);
} else {
$cache->put("queue_paused:{$connection}:{$queue}", true, $ttl);
}
}The 3. Connection Name ✅All methods now include connection name in the cache key: // Cache key format: queue_paused:{connection}:{queue}
queue_paused:redis:default
queue_paused:database:emailsThis ensures pausing Test Coverage: Thank you again for the thorough review! |
|
@yousefkadah , i still do not undertand use cases , where listing of paused queues can bee needed. The declared problem is that's no ability to pause certain queue. This problem is solved well without listing currrently paused queues. Take a look at mentionned earlier You're using cache locks to prevent race condition. Currently Queues does not depends on cache locks feature. Not all cache drivers (do not forget for custom ones) can provide locking feature. Pausing will fail with such drivers. Let's wait for @taylorotwell , only his opinion is significant here |
| { | ||
| $cache = $this->app['cache']->store(); | ||
|
|
||
| if ($ttl === null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you do not need conditions here, put method already implements forever if ttl=null
Problem
There is no way to pause a single queue in Laravel.
If you want to stop one queue, you must stop all queue workers — this halts all queues, not just the one you need to pause.
Solution
Add queue pause and resume functionality.
Workers will skip paused queues but continue processing other queues.
Artisan Commands
How It Works
Benefits
Example
This feature provides better queue control without stopping all workers.