[andybak]

> We use Celery for the following general cases:

>

> Communicating with 3rd party services (sending emails, notifications, etc.)

> Offloading heavier computational tasks outside the HTTP cycle.

> Periodic tasks (using Celery beat)

Sigh. No mention of the trade-offs. There's simpler ways to do all these things. Celery is a big complex beast and it always pains me to see it as the default suggestion for simple tasks.

[Spivak]

Because it’s mature, well integrated with Django and is a path so well-trodden there’s a McDonald’s on the way. Any possible use-case you can imagine for a job queue has been done in Celery and documented.

Celery being complicated is also entirely on the operational side, once you actually have Celery using it from within your app is simple enough.

Cron is awful for this use-case. You end up just inventing Celery but worse when you decide how your app and the cron scripts communicate. If you wanted just scheduled tasks but simpler use something like APScheduler.

[andybak]

> Celery being complicated is also entirely on the operational side, once you actually have Celery using it from within your app is simple enough.

Yeah and to some degree improvements in devops quality of life in the last few years has softened my view. (I used to mainly use Webfaction without access to apt-get and my own hand-rolled scripted deployment. Ugh...)

But I'd still usually prefer a pure-Python solution without additional persistent processes - assuming there is one and it's fairly well-documented. Huey is pretty good from recollection.

> Cron is awful for this use-case. You end up just inventing Celery but worse when you decide how your app and the cron scripts communicate. If you wanted just scheduled tasks but simpler use something like APScheduler.

I was advocating for something like this. There's django-cron etc which solves issues around communicating with scripts.

I do see both sides of the debate between "complexity" and "solves problems out of the box". I'm generally on the Django side when the flask vs Django discussion happens. There's always trade-offs.

[ccanassa]

I spent 3 years building a high scale crawler on top of Celery.

I can't recommend it. We found many bugs in the more advanced features of Celery (like Canvas) we also ran into some really weird issues like tasks getting duplicated for no reason [1].

The most concerning problem is that the project was abandoned. The original creator is not working on it anymore and all issues that we raised were ignored. We had to fork the project and apply our own fixes to it. This was 4 years ago so maybe things improved since them.

Celery is also extremely complex.

I would recommend https://dramatiq.io/ instead.

[1]: https://github.com/celery/celery/issues/4426

[hbrn]

> Because it’s mature, well integrated with Django and is a path so well-trodden

> Cron is awful for this use-case. You end up just inventing Celery

Isn't it the other way around?

Crons are way more mature, well integrated (mgmt commands don't require 3rd party modules), and extremely well trodden. Crons are super predictable, have sensible defaults and plenty of tooling. Which you will have to reinvent with Celery.

There are some benefits to programmatic crons, but the downsides are huge.

[aequitas]

A lot of people use Django with uWSGI, which also comes with queues, cron, workers, cache and lots more. I've been stuck with Celery on previous projects for reasons. But I've been dying to try out uWSGI's built in features for this. Hearing great things about it.

https://uwsgi-docs.readthedocs.io/en/latest/Spooler.html

https://uwsgi-docs.readthedocs.io/en/latest/Cron.html

https://uwsgi-docs.readthedocs.io/en/latest/Mules.html

https://uwsgi-docs.readthedocs.io/en/latest/Caching.html

[asalahli]

Fellow uWSGI fan here. Unfortunately uWSGI is now in maintenance mode, not because is complete which would've been fine, but because the maintainers are not able to dedicate time to it[0]

[nickjj]

A long time ago I wrote a blog post about Celery use cases at https://nickjanetakis.com/blog/4-use-cases-for-when-to-use-c....

It applies to Django, Flask or any Python system using it. All of it still applies today.

It covers a few use cases on the before vs after of using Celery and touches base on why I'd consider using Celery over other solutions such as async / await. The TL;DR is Celery brings a lot to the table around tracking and retrying jobs. It's also nice to separate your web and worker workloads into different processes since they have much different requirements usually.

[emptysea]

I think as a concept a task queue with some workers makes a lot of sense but having used Celery in production, it leaves a lot to be desired

We’ve run into various bugs and weird performance gotchas (like the workers prefetch jobs which is terrible if they aren’t all the same size)

[roflyear]

Agreed. I have had more luck writing my own "jobs" engine that does stuff that I need that celery doesn't have anyway (retries, some record of execution, rate limiting).

I'm sure if I really tried, I can get celery to be very reliable... but I never really got there.

Also for whatever reason I have NEVER been able to get celery to be reliable for its scheduling/cron stuff. It just starts to fail. I use this library for that, which I have never had problems with: https://schedule.readthedocs.io/en/stable/

[nickjj]

What type of situations did you run into while using it?

I've been using Celery for a long time in production now. Nothing crazy and it's a fairly basic set up of "execute job, thanks!" along with a beat server. Over the last 6-7 years an off the top of my head guess would be that there's been at least 5 million jobs processed. It's not huge volume when measured over years but it's been stable.

One server was running for 6 months without being updated. That's a Celery worker process running for ~180 days uninterrupted. It served hundreds of thousands of jobs without maintenance. A lot of these were pretty beefy tasks too like performing HTTP requests that got 400mb XML responses and then parsed them. I didn't even have things like `worker_max_tasks_per_child` set either.

[roflyear]

Celery has a lot of gotchas, but primarily the issue I have had is with resource consumption. Maybe it is a problem with my config, but I haven't had this issue with other solutions.

[notdarkyet]

Do you have additional examples of those simpler ways? I totally understand how Celery can be a hammer and everything is a nail type situation.

[nerdbaggy]

I am a big fan of Huey. And they have a Django module https://huey.readthedocs.io/en/latest/

[giancarlostoro]

I would just use serverless functions to achieve the same thing personally, if you're in the cloud already, chances are high you can trigger functions based on new records in a database or new file uploaded, or what have you. Then you don't need to import much outside of the serverless SDK which should typically be pretty minimal.

That's how I did a timed function for a Django project we were hosting in Azure anyway.

[roflyear]

Serverless functions are good for some stuff: clear this cache on this schedule, run this task when XYZ happens, etc..

But that is not the popular usecase for celery. Often you want "some code" (that you likely already have written) to be executed async. Sure, you can create a public interface for "some code" then write a record, that the serverless function is looking for, that then calls back to that interface (but is it a web interface???? then you have a problem where if the job takes too long to complete, what about http timeouts ... ) and now you're really creating a big circle for something that should be simple: execute some code outside of the request (send an email, hit an api, whatever).

Serverless functions really shouldn't contain much logic either, because it's too complicated to test.

[giancarlostoro]

Ive used serverless in this way as well when it was a long running process, basically the end-user needed to upload a Shapefile, and some of them can be quite large, so I kicked off an Azure Function once the file was uploaded to parse the file in the background. If it's something that will halt the browser when it needs to run in the background instead, that's where I'll use Serverless if it makes sense. I'm not fond of having my web server running things in the background it ruins my mental model of the web.

[roflyear]

> I'm not fond of having my web server running things in the background it ruins my mental model of the web.

Well yes, and you run into other problems as well (now if your process dies or you deploy or something you have to be careful to not kill running jobs).

How much of your logic is in the azure function?

[giancarlostoro]

Only things that needed to be done behind the scenes that could take a long time, like we would take JTIFFs that could be gigs of data, and analyze them and create variations of the same image. You don't want someone who just uploaded a 1GB file waiting for you to also analyze it as well.

[j4mie]

https://github.com/dabapps/django-db-queue

(I am the author)

EDIT: oh hi, Andy :)

[nicolaslem]

Since everyone suggests alternatives to Celery, may I plug my own healthier celery?

https://github.com/NicolasLM/spinach

[69Represente]

I have a bug with celery i can't solve.

when I send an async job that get data from various APIs and write all in a DB, in the case of lot there is lot of data, the celery task finish properly to but my flask app becomes unresponsive. I have to restart flask to get back to a normal state.

Anyone would know where I should check?

[roflyear]

sounds like a resource issue, maybe you are opening application contexts and not handling them properly?

[collyw]

Django subreddit. Someone asks for a way to run a simple cron job once a week. Everyone jumps in and shouts "Celery". I chip in and say, nah, just set up a cron job, it will be simpler. Get downvoted by the hivemind.

[andrewingram]

I've been using the beta of Temporal's Python SDK with Django, and aside from some minor teething issues, it's very nice.

[hdjjhhvvhga]

What would be your first choices for each of the above?

[true_religion]

My choice for periodic jobs is cronjobs.

[VWWHFSfQ]

Do you mean a cronjob that calls a Django manager command to do the work? Or invokes an API method? From my experience cronjobs have a lot of downsides as well. They're great for doing tasks local to the server the job is running on. Not so great for the kinds of tasks (periodic or transactional) that Celery/a real queue is designed for.

[danjac]

Crons are fine for running Django commands that are just talking to the same services as your web views (databases, cache, email services etc). It should be fine to let them run in their dedicated VM/containers, using Ansible etc to keep the crontab configurations in the repo.

Celerybeat has the advantage of better visibility e.g. you can configure them in the Django admin and check when they are running. If you are not using Celery though and your needs are simple, it's easier to just use plain crons.

[true_religion]

I use them for invoking django commands on the same server. I do use celery for transactional jobs though. It’s only periodic jobs that get called with cron. For the context, I do this on small web apps with less than 20 dedicated servers (real servers not VPS), so there is a “manager” server that does nothing but run periodic tasks, management interventions, and cleanup operations.

[andybak]

Yeah. Or a simple cron wrapper like django-cron to get the best of both worlds.

For background tasks - you can just spawn a background process and keep a simple status table in the db so the main app can check if it's completed (assuming you even need that)

And for task queues that can handle the traffic most sites will need there's things like django-huey.

[VWWHFSfQ]

> For background tasks - you can just spawn a background process and keep a simple status table in the db so the main app can check if it's completed (assuming you even need that)

I don't know if making my own bespoke queue system is a great idea. It seems simple enough, but it gets so much more complicated once you start seeing issues with it. Orphaned task processes sticking around on the server forever, concurrency control, error handling, etc. I'll pretty much always just use celery and not have to worry about it.

[andybak]

I'd never argue for building your own - just for using something simpler than Celery

It's not so bad now as CI/CD, Docker etc have made complex deployments easier to handle. But back when I was wrestling with Django simply deploying Celery on a new host could easily waste an afternoon and all those dependencies made me very nervous about the overall complexity.

I still weigh carefully anything that adds another long-running process or non-Python dependency to my sites.

[roflyear]

This is good for really simple scheduling of stuff: https://schedule.readthedocs.io/en/stable/