Celery Beat not dispatching Tasks after upgrade form `2.8.0` to `2.8.1`

[FabianClemenz]

I have updated to version 2.8.1 this morning and celery beat stopped dispatching tasks since then. I'm using the database scheduler and i think this is related to the change of using server timezone.

We have timezone UTC on the server but Europe/Berlin in our django application. Also setting timezone to Europe/Berlin on the server stops the tasks from beeing dispatched.

#886

[yanivtoledano]

We see something similar on our end.

Basically, ever since our upgrade to 2.8.0 (and now 2.8.1) we have intermittent failures where beat tasks are no longer triggered. The only way to trigger them is to trigger a new build on our servers for some reason (this was never the case before 2.8.0/1).

If I'm not mistaken, the original issue comes from this pr #879. Can't we simply revert that pr (and the various fixes since that pr) and publish a 2.8.2?

To be clear - having upgraded to 2.8.1 has not fixed the issue.

[FabianClemenz]

On my side i don't know why it works with 2.8.0 before upgrading and now after downgrading doesn't work...

[vzii]

Yes, I have the same issue

[auvipy]

we got a new PR #896, not sure if it is addressing your issue? can you check?

[FabianClemenz]

@auvipy are there anywhere docs on how to correctly setup and test celery beat locally for contributing?

[auvipy]

you can check this https://github.com/celery/django-celery-beat?tab=readme-ov-file#developing-django-celery-beat also we use tox for test automation

[FabianClemenz]

Ok thanks, when I get time I’ll check it. I‘ll try to manually get the wrong times so I can check where the error happens

FYI:
On one of our servers I manually reverted the tasks as you mentioned in your docs when changing timezone. This morning the tasks ran, but with utc instead of Europe/Berlin

[alirafiei75]

The only way to trigger them is to trigger a new build on our servers for some reason

@yanivtoledano Can you explain this more?

[alirafiei75]

we got a new PR #896, not sure if it is addressing your issue? can you check?

I do not think that this new PR is related to this issue. Also do not think anyone is using sqlite on production, cause it is a bad practice.

[yanivtoledano]

@alirafiei75 - answers below:

Can you explain this more?

Our Celery Beat tasks run fine throughout the day (cron jobs are triggered as expected). Every night though, the tasks stop triggering anywhere from 2-4 AM in the morning UTC. We know this because we have tasks running every 5 minutes which suddenly stop. This issue did not exist before version 2.8.0 (we never had any issue with Celery Beat skipping/stopping). The only way for tasks to start running again in production is to rebuild (e.g., by pushing a commit to prod).

I do not think that this new PR is related to this issue.

That's my mistake then (we run Postgres in production). Basically, before 2.8.x we never had issues with Celery Beat. Ever since upgrading we have daily issues. We can't downgrade to 2.7.x because we're on Django 5.2.x.

[alexbehl]

FYI, our task scheduling also broke after the latest dependency update. However, it does not seem to be directly related to Celery Beat. Rather, updating @click (Celery dependency) to version 8.2.0 seems to be the cause. Pinning it to 8.1.8 resolved the issue.

[jp26jp]

@yanivtoledano i'm having the exact same issue. Interval tasks get picked back up, but all early morning crontab tasks have stopped running. This started after upgrading from 2.7.0 to 2.8.0.

After downgrading back to 2.7.0, the issue was fully resolved.

[yjchen-tw]

FYI, our task scheduling also broke after the latest dependency update. However, it does not seem to be directly related to Celery Beat. Rather, updating @click (Celery dependency) to version 8.2.0 seems to be the cause. Pinning it to 8.1.8 resolved the issue.

We have environment using click 8.1.8 with django celery beat 2.8.0, still have the same issue.

[Gornoka]

@yanivtoledano i'm having the exact same issue. Interval tasks get picked back up, but all early morning crontab tasks have stopped running. This started after upgrading from 2.7.0 to 2.8.0.

After downgrading back to 2.7.0, the issue was fully resolved.

We had downgraded from 2.8.1 to 2.7.0 as well, however that did not resolve it for our interval tasks ( downgrading click helped tho)