Hacker News new | ask | show | jobs
by lizthegrey 1665 days ago
> What was the reason to stick on 0.10.0 for so long?

Aforementioned self-packaging, we were mangling the .tar.gz files into .debs, and we had to remember to update the debs and then push them out onto our systems, instead of just using Apt. Thus why Confluent's prebuilt distro helped a lot! But also the team was just _afraid_ of Kafka and didn't want to touch it unnecessarily.

> I see you like Cruise Control in the Confluent Platform, did you try it earlier?

We definitely should have. We tried Datadog's Kafka-kit but found adapting it to use Wavefront or Honeycomb Metrics products was more problematic than it needed to be.

> Tiered storage is slowly coming to FOSS Kafka, hopefully in 3.2.0, thanks to some very nice developers from AirBnB. Credit to the StreamNative team, that FOSS Pulsar has tiered storage built-in.

Yeah, we're glad the rest of the world gets to have it, and also glad we paid upfront for Confluent's enterprise feature version to get us out of the immediate bind we had in 2020. Those EBS/instance storage bills were adding up fast.

2 comments

The irony of Honeycomb using an open source tool from Datadog is not lost on me =)
This is one of the major benefits of open source. You can share engineering resources, even with your competitors, for things that aren't your bread & butter.
Yup! In fact, we collaborate with Datadog, New Relic, Splunk, Lightstep, et al on OpenTelemetry (https://news.ycombinator.com/item?id=28997275 / opentelemetry.io)
stand on the shoulders of giants!
> we paid upfront for Confluent's enterprise feature version to get us out of the immediate bind we had in 2020.

Definitely agree it's an essential feature for large datasets - in the past I've used Kafka Connect to stream data to S3 for longer term retention, but it's something else to manage, and getting data back into a topic if needed can be a bit painful.

getting to just use the same consistent API without rewriting clients was AMAZING.
This always reminds me of this thing I’ve written some time ago : https://gruchalski.com/posts/2021-04-02-kafka-infinite-reten...