[keosak]

Points 1 and 2 are only valid because the Celery database backend implementation uses generic SQLAlchemy. Chances are, if you are using a relational database, it's PostgreSQL. And it does have an asynchronous notification system (LISTEN, NOTIFY), and this system allows you to specify which channel to listen/notify on.

With the psycopg2 module, you can use this mechanism together with select(), so your worker thread(s) don't have to poll at all. They even have an example in the documentation.

http://www.postgresql.org/docs/9.3/interactive/sql-notify.ht...

http://initd.org/psycopg/docs/advanced.html#async-notify

[denibertovic]

It is true that Postgres supports Pub/Sub but unfortunately the Celery broker driver does not take advantage of this. It would be great if we could get support for it. Nevertheless, just because it has pub/sub doesn't mean it's a full AMQP implementation. Also, there's the fact that most amqp solutions are in memory, wheres a database is on disk... also has it's costs.

[denibertovic]

Anyone that's interested in Postgres's pub/sub might find this useful: https://denibertovic.com/talks/real-time-notifications/#/

Just slides though. Haven't gotten around to writing a post about it yet.

[waffle_ss]

Yep, in Ruby there is a background processing gem built around this: https://github.com/ryandotsmith/queue_classic

Unfortunately if you're using JRuby you can't benefit from this, as the Postgres JDBC driver does polling.

[ddorian43]

but each worker will occupy a connection/session/process which are heavyweight ?

[denibertovic]

It is true that each worker will use up a connection. Not sure how heavy weight it is, depends on your setup and use-case I guess.