Hacker News new | ask | show | jobs
by Spivak 1250 days ago
Because it’s mature, well integrated with Django and is a path so well-trodden there’s a McDonald’s on the way. Any possible use-case you can imagine for a job queue has been done in Celery and documented.

Celery being complicated is also entirely on the operational side, once you actually have Celery using it from within your app is simple enough.

Cron is awful for this use-case. You end up just inventing Celery but worse when you decide how your app and the cron scripts communicate. If you wanted just scheduled tasks but simpler use something like APScheduler.

3 comments

> Celery being complicated is also entirely on the operational side, once you actually have Celery using it from within your app is simple enough.

Yeah and to some degree improvements in devops quality of life in the last few years has softened my view. (I used to mainly use Webfaction without access to apt-get and my own hand-rolled scripted deployment. Ugh...)

But I'd still usually prefer a pure-Python solution without additional persistent processes - assuming there is one and it's fairly well-documented. Huey is pretty good from recollection.

> Cron is awful for this use-case. You end up just inventing Celery but worse when you decide how your app and the cron scripts communicate. If you wanted just scheduled tasks but simpler use something like APScheduler.

I was advocating for something like this. There's django-cron etc which solves issues around communicating with scripts.

I do see both sides of the debate between "complexity" and "solves problems out of the box". I'm generally on the Django side when the flask vs Django discussion happens. There's always trade-offs.

I spent 3 years building a high scale crawler on top of Celery.

I can't recommend it. We found many bugs in the more advanced features of Celery (like Canvas) we also ran into some really weird issues like tasks getting duplicated for no reason [1].

The most concerning problem is that the project was abandoned. The original creator is not working on it anymore and all issues that we raised were ignored. We had to fork the project and apply our own fixes to it. This was 4 years ago so maybe things improved since them.

Celery is also extremely complex.

I would recommend https://dramatiq.io/ instead.

[1]: https://github.com/celery/celery/issues/4426

> Because it’s mature, well integrated with Django and is a path so well-trodden

> Cron is awful for this use-case. You end up just inventing Celery

Isn't it the other way around?

Crons are way more mature, well integrated (mgmt commands don't require 3rd party modules), and extremely well trodden. Crons are super predictable, have sensible defaults and plenty of tooling. Which you will have to reinvent with Celery.

There are some benefits to programmatic crons, but the downsides are huge.