| Hello HN, we’re open sourcing a new library called Quix Streams (https://github.com/quixio/quix-streams)—a high-performance Python and C# library that processes time-series telemetry data with message broker such as Apache Kafka, instead of a database. We’re a team of engineers who all worked together at the Formula 1 McLaren pit wall. We struggled to get wanted we wanted out of existing stream processing tech, so we built our own platform called Quix. Although our original requirements were pretty niche (process over 60k data points per second), our solution is designed to make life easier for anyone who needs to process high volumes of time-series data. We also wanted it to be as accessible as possible for data scientists and ML engineers (e.g. those who run a lot of physics models on sensor data). This means it’s got a lot of built-in convenience functions such as native DataFrame support and easy-to-use buffers for rolling window calculation. The main differentiator though, is making it as performant as possible—we’ve added features that make it easier take full advantage of Kafka’s parallel processing capabilities while keeping relevant data together and ordered correctly. So even if you’re not doing real-time ML on Formula 1 telemetry, we hope that its simplicity and performance will make you more productive and we’re excited to see what other use cases you find for it. More info: —https://github.com/quixio/quix-streams —https://quix.io/blog/open-sourcing-quix-streams —https://docs.quix.io/sdk-intro.html#library-features * Full disclosure: the library started as a companion to our commercial platform http://quix.io but you can use it with any Kafka installation you like. |