[royjacobs]

What was the reason for not contributing a "reactive" API to the existing C client?

Especially because now you have a maintenance burden on your own client which, ironically, performs quite a lot slower than the C client although that was the major concern for not using it in the first place!

[hackmanytrades]

That's a great question.

Yes, Pony Kafka is currently slower than the C client. But it is also almost completely untuned as of right now. We expect there is a lot of low hanging fruit on that front that will give us significant gains.

There is also the secondary concern regarding the thread pools internal to Pony and librdkafka. We've seen first hand how CPU cache invalidation can impact performance so we are very aware of the potential negatives if the Pony and librdkafka threads ever end up fighting with each other over the same CPU resources.

[royjacobs]

Sorry, that was my question-- was there a way to avoid using librdkafka's threadpool and essentially use it as a 'dumb' client, moving all the async stuff to your Pony actor layer?

[hackmanytrades]

From what I understand of librdkafka, there's no easy way to disable the internal thread pool it uses.

I'd imagine that the internal thread pool for sending/receiving data from Kafka is as core to librdkafka as the internal thread pool for running actors is to Pony and trying to remove or disable either of them would be a large undertaking.

[vbernat]

The existing API already features ability to integrate into any event loop. See rd_kafka_queue_io_event_enable() to provide a file descriptor to be notified of incoming data.

[sidlls]

They made a decision some time ago that they were a Pony shop. Adding something written in a different language would from their perspective (perhaps) be the additional maintenance burden.