"Why we wrote our Kafka Client in Pony (wallaroolabs.com)"
I love the fact that the words "Kafka," "Pony," and "Wallaroo" are all together in one sentence, and not as part of an elaborate joke. I love even more the fact that Serious Business Executives will have to use these words to discuss useful technology. Awesome names.
First - I spent a good amount of time with Pony attempting to make something similar to Wallaroo (on a much smaller scale), and it seems I hit the same problems as you re:FFI. Does it worry you that creating idiomatic C wrappers in Pony feels 'wrong'? Are there ways to make it better? After all, young languages especially are very dependent on wrapping C libs for basic functionality.
Second - from what I can tell, Wallaroo Labs are now one of the main contributors to Ponylang. How are your experiences from writing Wallaroo affecting language design and direction?
Re: FFI. In general, using the C FFI and creating C wrappers is not a major issue (aside from them possibly not being idiomatic) and, as you mentioned, required for a language as young as Pony. In fact, Pony Kafka internally relies on a number of C libraries via FFI. However, for the Pony Kafka use case, performance was a key driver. This meant that the risk of contention between the librdkafka thread pool and the Pony thread pool was one that had to be taken into account for long term performance goals. Same regarding the polling nature of getting data from librdkafka. Neither of these would have been a major issue had performance not been a high priority for Wallaroo. In regards to the idiomatic C wrappers, I think it's possible to wrap C libraries in an idiomatic way. It's not easy though because it's hard to figure out the right abstractions in Pony and map them to the C functionality. I don't think that's a problem though because not everything has to be idiomatic from day one. Over time, the C wrappers can be improved to be more idiomatic and/or phased out via native implementations.
Re: Ponylang direction. My personal experience on this is that the Pony community has been very receptive to our ideas. I've mainly focused one the Pony runtime side of things and less on the compiler/language design side of things though.
I love the fact that the words "Kafka," "Pony," and "Wallaroo" are all together in one sentence, and not as part of an elaborate joke. I love even more the fact that Serious Business Executives will have to use these words to discuss useful technology. Awesome names.