[random023987]

> I would love is an event store

You know you can put a key with a null value onto a topic. If your backend object store gives you data by key, this should give you exactly what you want.

Later on, you can put a small subset of the data in value to assist with filtering or caching (if you want).

[derefr]

I'm not clear on what you're trying to suggest, here. I would like a system that presents itself the same as a Kafka broker, to both producers and consumers (who are just working with regular messages and topics, not object-handles), but where the MQ broker they're talking to is internally just a shared-nothing stateful write-through WAL proxy for an object store. (Sort of like how, with actual logging, you could think of the local rsyslog daemon on an instance as a "shared-nothing stateful WAL-writeback-caching proxy for an object store." Except rsyslog lives on the producer, whereas this write-through WAL proxy would live on the broker.)

In such a design, "partitions" and partition ownership would only apply to the newest segment of the log; all older segments would be globally-available, because as soon as a log-segment is finalized, it would get pushed to the object store and become an object.

Every node would then be doing Hierarchical Storage Management, with a local LRU cache of segments, populated just-in-time in response to requests, by the object store. (This is what I meant by the comparison to Datomic's storage architecture.)

A lot of the technical parts of this could be achieved by just FUSE-mounting an S3 bucket to a directory on each Kafka node, and convincing the node to move log segments into that directory once they're done. But Kafka would still be wasting time and resources replicating segments between nodes that already now "have" them; and also, without further hacking, nodes wouldn't realize that they now also "have" 99% of the segments of every log potentially-available to serve. A system that was built this way from the ground up would end up a lot simpler than Kafka.