I think it'd be very easy to write your own. I used postgres subscribe/listen built in combined with a database table to get a distributed message system.
Writing a distributed, scalable system is really hard, and beyond the API, that is the real value for kafka
It's relatively easy -- removing any networking requirements drastically simplifies the problem. There's still some non-trivial bits that vary depending on granularity for concurrency.
It's a weekend project to demonstrate the concept, maybe a few weeks to really flesh it out and iron out quirks. I imagine if you're willing to use sqlite as a backend for persistence, it gets a bit easier.
You may be right. However, consider the directorial perspective.
You have employees - you try to get and retain the best talent you can. However, every human has strengths and weaknesses, and these may not all be fully visible to you.
Rolling your own vs buying off the shelf is a gamble on future outages.
Will a third-party support and fix the issue, or have a strong community that can help you work through the issue?
If your best engineer builds something that works for long enough to become entrenched, but then carks it, will your best engineering talent be able to resolve the issue? If your rockstar quits, does the team have to pick through the halls of Cthulhu? Does your organisational ignorance of kernel networking suddenly become painfully apparent?
Remember, you need to be twice as clever to debug your code than to write it...
One of my desires would be for it to be persistent. Hopefully with the option of different storage tiers, so as logs became older they could be moved to less costly medium and transparently fetched when requested.
Having an event sourced system doesn't make much sense unless you maintain messages from the start of the system. You can snapshot state and resume in order to quickly rebuild from a known good state. That doesn't help if there was a logic error corrupting every state from the start, and a full rebuild is required.
I'm unsure how redis streams behave with regard to cache eviction, nor am I familiar enough with rabbitmq to comment on it's behavior. It's been 10 years since I used either, and at the time neither were good solutions for a log based system.
Redis doesn't do key eviction by default. Everything lives forever, unless you tell it otherwise. Of course, it is recommended to turn on some form of persistence (and configure backups) so that things don't disappear if the server restarts.
Writing a distributed, scalable system is really hard, and beyond the API, that is the real value for kafka