[mac01021]

As someone who has worked on building a custom consumer API for kafka (in a distant past, before the 9.0 Java consumer came out) I am curious about how much engineering effort this required.

I can see from github that the project is about 16k lines of code. I wonder how many developers worked on it, how much of each of their time it required, how many false starts in the architecture of the library had to be abandoned...

[hackmanytrades]

99% of Pony Kafka is written by me (for better or worse). I've been working on it since about May 2017 off and on with it being my primary focus for the majority of that time. However, due to working arrangements and other commitments, I've only spent about 12 or so weeks of time on it (where 1 week is equal to 5 days and 1 day is equal to 8 hours).

There have been a few iterations on the abstractions and API of the library but the majority of the architecture has been the same from the initial design sketch. I started by envisioning the features the library needed for end users and also internally in order to fully take advantage of Pony's actor concurrency model. From there I worked out the data and functionality ownership of the various bits (i.e. which actor does what and why). Lastly, I ran it by a couple of folks here at Wallaroo Labs to make sure I wasn't making any obvious mistakes.

The biggest change so far has been caused by building in the leader failover handling in relation to the data/responsibility ownership transferring from one actor to another. That's not entirely completed yet but it has mostly been an internal change. The end user API has also changed, but that has mostly been about fixing abstractions and/or data ownership issues.

I'm sure there will be additional changes as I have time to go through to fix abstractions and add in other features. Dynamic configuration changes, exactly once semantics, and group consumer functionality are all likely to impact the end user API along with requiring internal changes.