Hacker News new | ask | show | jobs
by Joeri 1831 days ago
For me a perfect use case is sensor data processing. I've been involved with two independent sensor data platforms that used kafka as the backbone. Sensor data is persisted unprocessed to raw kafka topics, and then gradually deduplicated, refined, enriched and aggregated across multiple stages until finally a normalized, functionally sharded and aggregated view of the data is stored in databases for dashboarding.

It is easy to scale horizontally to massive volumes of data, and any issues in the processing pipeline can be fixed without losing any raw sensor data (restarting the consumers from the last known valid point).