[pas]

Kafka is at least a bit simpler than RabbitMQ, though both are very square-ish shaped pegs and are usually forced into very round-ish holes. People regularly think they need a message queue, when they really need a job queue, or message bus. Or even all three, but then they try to hack everything on top of Kafka (or RMQ) ... which can be done, of course, but "results may vary".

For tracking state at scale (but still per-job, per-thing) a Cassandra-like system works best (but preferably a better implementation, eg. SkyllaDB or AeroSpike or some other KV store).

[CharlieDigital]

> People regularly think they need a message queue, when they really need a job queue, or message bus

This is one of the reasons I am a really, really big fan of Google Cloud's Task Queues. It allows the stupidest, simplest temporal execution of HTTP invocations.

Currently working on a project in AWS and it's stunning how complicated it is to achieve the same simple need of "I want to execute this HTTP call at this time in the future". It's either AmazonMQ -- using either ActiveMQ or RabbitMQ with plugins -- or hacking around SQS's 15 minute delay limit. In our case, we are going to end up wrapping our messages in an envelope with a delivery time and if it hasn't met the delivery time, we put it back into SQS.

GCP is highly underrated for how it simplifies control over execution of code. Pub/Sub and Task Queues both have HTTP delivery built in. Couple that with Google Cloud Run and it is a recipe for building almost any type of execution model with much less complexity and overhead

[grncdr]

In case you want to avoid an extra envelope, you can also add custom headers to SQS messages. This can be handy if you want to implement that delay hack without parsing message bodies.

[davydog187]

Lol to “ Kafka is at least a bit simpler than RabbitMQ”. I’m sorry, what universe does this statement live in?

[Spivak]

True, it’s actually a lot simpler than RabbitMQ. People seem to assume “giant ball of enterprisey Java means” that the experience using it will be complicated. Kafka is extremely reliable, simple to cluster, battle tested (I haven’t hit an actual bug in Kafka in ages), the self-healing is turnkey, and has way stronger guarantees for clients.

Where it bites people is that it’s not a queue and scaling is harder than just add more consumers.

[rawgabbit]

I am curious if anyone has used confluent.io (kafka as a service). Or is it too cost prohibitive?

[d8tltanc]

Java is a bit simpler than erlang

[dariusj18]

I looked it up but couldn't really find information, what would you say is the difference between a "message queue", a "job queue" and a "message bus"?

[pas]

I just throw them on the wall based on my experiences, maybe they have some agreed upon precise definitions, but I'm not aware :o

message bus: firehose of events, (by default) no ACK. usually multi producer multi consumer. (see also DBus which is more of an RPC layer + service discovery + pub/sub via event listeners)

message queue: usually between components, ACK, but no selective ACK, backpressure, might even have support for "dead letters" (letters not ACKed by any consumer)

job queue: selective ACK, retry, etc.

(there's also the "enterprise service bus", which is similar, but mostly implemented on things like IBM MQ)