[koolba]

I've wondered why there isn't a "big player" in the cloud space for this. Felt like a hole.

My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub/sub. Ditto for SQS for externally triggered eventing.

[bjt]

> For smaller operations Redis is more than enough for pub/sub.

I didn't find that to be so at my last job, one of those smaller operations.

With Redis you're forced to pick between two severely constrained options:

1. Use PUBLISH/SUBSCRIBE. This is nice if you want to have several listeners all receive the same message. But if a listener is down, there's no way for it to recover a message that it missed. If there is no one listening, messages are just dropped.

2. Use LPUSH/BRPOP. This is nice if you want to have several workers all pulling from the same queue, but isn't sufficient if you want to have several queues streaming from the same topic. (E.g. one listener is responsible for syncing to ElasticSearch and another one is syncing to your analytics DB.)

I strongly prefer RabbitMQ. Its model of exchanges and queues supports mixing and matching these semantics much more flexibly.

[manigandham]

Agreed, and this is because Redis is a database first, with some pub/sub and nice lists functionality. RabbitMQ is a proper message queue (mq) which provides the necessary features for bigger applications.

However RabbitMQ is also pretty fragile and terrible at scaling. NATS.io is another system that's got the messaging right and is working on persistence soon.

[kinkdr]

How stable is RabbitMQ? I've been looking into moving from away from redis pub/sub for a bit now.

[manigandham]

RabbitMQ is ok in single server and has lots of flexibility but struggles at high throughput ( > 100k/sec) and the clustering setup is not great. There are also lots of edge case bugs.

If you don't need persistence, look at using nats.io which is a much more stable and reliable pub/sub system. You can build persistence on top of it or wait a few months until they finish their new project STAN.

[kinkdr]

Thanks! 100k is far more than I need, but I couldn't find something that would fit exactly my needs, so I ended up rolling my own.

[alexatkeplar]

IBM Bluemix has been offered "pure" hosted Kafka as their MessageHub product for a little while: https://developer.ibm.com/messaging/message-hub/

MessageHub's lead engineer Oliver Deakin gave a talk at Unified Log London recently where he explained how MessageHub was architected under the hood, was super-interesting. Slides available from here: http://www.meetup.com/unified-log-london/events/229693782/

[rhodin]

Not a "big player" (yet), but we've been offering Apache Kafka as a Service since June 2015: www.cloudkafka.com.

[amock]

Depending on what you mean by "this" there are offerings by the big players. Google has Cloud Pub/Sub and AWS has Kinesis in addition to SQS, so two of the big players do have offerings. I'm not familiar enough with Azure to know what it has.

[koolba]

By "this" I meant a managed Kafka cloud offering. I generally a fan of these types of services as there isn't as tight a binding as proprietary ones. Migrating from Heroku Postgres to RDS or self hosted is well defined. Ditto for Redis migrations.

SQS, Kinesis, and other proprietary ones not so much. You can insulate your code base but if you're really going to leverage the ecosystem of those services then you're going to be stuck there. That's why I find something like this interesting. The "out" is there so it makes it easier to accept getting in.

[manigandham]

There really isn't much lock-in when it comes to event logging systems. Just change the interface your code uses to whatever service you need. There might be a little refactoring to handle topics in the different ways but it's all ultimately the same thing.

Since logging by nature offers asynchronous processing, you can migrate your publishers first and then the consumers without any downtime.

[manigandham]

Azure has Event Hubs that are very similar to Kinesis/Kafka.

https://azure.microsoft.com/en-us/services/event-hubs/

They also have simpler Queues and Service Bus for RPC/lightweight message handling.