fix: simplify task fetching by removing crontab exclusion #956

[ncimino]

Removed crontab exclusion logic from task fetching. Filters were created to limit window to +/-2 hours for crontab, but tasks are never refetched - must either implement refetching protocol or (as implemented here) remove the filter entirely.

[auvipy]

can not verify this as there is no appropriate test cases. and the CI is failing. i think it would be better to implement improved protocol

[ncimino]

can not verify this as there is no appropriate test cases. and the CI is failing. i think it would be better to implement improved protocol

@auvipy / @alirafiei75 any suggestions on how/when the tasks should be getting refetched/parsed from the DB? The code obviously expected this behavior, but it is not happening. I'm also curious if there are other tests that we expected to cover this code originally?

[kamyanskiy]

can not verify this as there is no appropriate test cases. and the CI is failing. i think it would be better to implement improved protocol

I think that mechanism with sliding window is not bad, we just need to force this sliding every hour or less.
In 2.8.1 was added code that seems forces full reload every 5 minutes. So it seems its not a problem anymore

django-celery-beat/django_celery_beat/schedulers.py

Lines 519 to 531 in 6427286

	# Force update the schedule if it's been more than 5 minutes
	if not update:
	time_since_last_sync = (
	current_time - self._last_full_sync
	).total_seconds()
	if (
	time_since_last_sync >= SCHEDULE_SYNC_MAX_INTERVAL
	):
	debug(
	'DatabaseScheduler: Forcing full sync after 5 minutes'
	)
	update = True
	self._last_full_sync = current_time

[auvipy]

but we need a proper solution with appropriate tests

[ncimino]

That 5 minute force refresh is not happening for crontab. Some code runs, but the reload never rereads/refreshes from the database.

This code you are referring to as the fix has been there since before v2.8.1 and the issue presents in v2.8.1:
https://github.com/celery/django-celery-beat/blame/v2.8.1/django_celery_beat/schedulers.py#L515-L527

As @auvipy has indicated, I think testing (if representative of client crontab use case) will be quite revealing.

[kamyanskiy]

@ncimino @auvipy I would say that on 2.8.1 its not an issue anymore - just look at my logs - I showed that its work here #956 (comment)

so sliding window works (at least with celery 5.5.3), you can see how it forces reload crontab schedules every 300sec and it works as well, my tasks those were planned out of sliding window, finally loaded in memory when window overlaps it, so I can see crontab tasks in my schedule in (-2 +2h) period. Looks good.

[auvipy]

@ncimino @auvipy I would say that on 2.8.1 its not an issue anymore - just look at my logs - I showed that its work here #956 (comment)

so sliding window works (at least with celery 5.5.3), you can see how it forces reload crontab schedules every 300sec and it works as well, my tasks those were planned out of sliding window, finally loaded in memory when window overlaps it, so I can see crontab tasks in my schedule in (-2 +2h) period. Looks good.

thanks for verifying