Hacker News new | ask | show | jobs
by mannyv 1361 days ago
You can spew messages into sqs then have a lambda on sqs sending to pg.

Funny, we were going to use kafka too, but just sending to mysql worked just fine <shrug>. One day that will change, obviously.
1 comments
eurasiantiger 1361 days ago
It works fine on the happy path, but if mysql goes down, you lose things. Having a distributed, resilient queue has availability benefits.
link
pritambaral 1361 days ago
> It works fine on the happy path, but if mysql goes down, you lose things.

Why do you have to lose things if the DB goes down? Agreed, untuned & unconfigured MySQL (and MongoDB) out-of-the-box can lose things due to bugs and design issues, but that is the case even when they are running. However, DBs, in general are made precisely for the purpose of safely storing things and not losing them.

OTOH, the number of Kafka setups I have seen that'd lose things when something goes down ... maybe this is not a guaranteed win for the Kafka side of arguments.

> Having a distributed, resilient queue has availability benefits.

High availability is not a function exclusive to Kafka. On the other hand, there's some functions that may come in handy to use in a queue that Kafka simply cannot provide, but DBs can. Off the top of my head: ACID, instant scalability (both up and down) of consumer groups, and the sheer flexibility (and power) that comes with a DB in general.

----

Overall, there's some merits to using a distributed log as a message queue, sure, but there are also merits to using a DB for that.

link
EdwardDiego 1360 days ago
> OTOH, the number of Kafka setups I have seen that'd lose things when something goes down ...

Was the data actually acked across all replicas by Kafka? If so, very interested in how it was lost.

link
manv1 1360 days ago
That's why we have an RDS cluster. We're not multi-region because, frankly, if the region barfs we can blame it on Amazon. And in any case we'll still deliver just fine as long as cloudfront keeps going.
link
manv1 1360 days ago
Also, we use SQS as an intermediary between our logging and mysql, so even if mysql goes down SQS will keep things for...a while.
link