[richieartoul]

(WarpStream founder)

It guarantees correct ordering inside a partition for all acknowledged messages regardless of which batch they originated from. We don't synchronize clocks, the agents call out to our cloud metadata store which runs a per-cluster metadata store that assigns offsets to messages at commit time. Effectively "committing" data involves two steps:

1. Write a file to S3 2. "Commit" that file to the metadata store which will then assign the partitions at commit time 3. Return the assigned partitions to the client

[wdwind]

Then the order is on batch level? Say batch1 is committed with a smaller timestamp than batch2, then all messages in batch1 are considered prior/earlier than any message in batch2?

[richieartoul]

It’s getting late and I’m not 100% confident I’m sure what you’re asking, but I believe the answer to your question is yes. When you produce a message/batch you get back offsets for each message you produced. If you produce again after receiving that acknowledgement, the next set of offsets you receive are guaranteed to be larger than any of the offsets you received from your previous request.