Hacker News new | ask | show | jobs
by nullwasamistake 2589 days ago
Sounds like your org used Kafka for event sourcing. This is almost always a bad idea, event sourcing and aggregate reconstruction is a nightmare IMO.

Kafka used as a pure FIFO cache for regular CRUD endpoints works fine
2 comments
pablasso 2589 days ago
Event sourcing was one of LinkedIn use cases when they created it, Kafka is fine for all logging needs.
link
josephg 2589 days ago
Yes; they did. It worked pretty well actually.

Why do you think it’s a bad idea? Most of the arguments against event sourcing that I’ve read seem to be “yes but the tooling isn’t very good”. That might be true, but maybe we solve that problem with more investment into event sourcing; not less of it.

link
nullwasamistake 2589 days ago
TLDR the tooling is so bad it's basically impossible to run at scale. I worked for a company that tried. Maybe on a small scale it's fine, but replays and storage of past events takes insane amounts of space at high event rates. To the point that storage costs and replay times became a real problem. (Many terabytes and days)

I also don't think it's a great idea in general. The event stream directly replicates a DB commit log, and the aggregates your tables. It's building your own database.

We had to throw a year's worth of work away at the end so I'm fairly biased against trying it until the ecosystem is better.

link