[justicezyx]

I led the storage engine prototyping for Kinesis in 2012 (the best time in my career so far).

Kinesis uses Chain Replication, a dead simple fault tolerante storage algorithm: machines formed a chain, data flow from head to tail in one direction, writes always start at head, and read at tail, new nodes always join at tail, but nodes can be kicked out at any position.

The membership management of chain node is done through a paxos-based consensus service like chubby or zookeeper. Allan [2] (the best engineer I personally worked with so far, way better than anyone I encountered) wrote that system. The Java code quality shows itself after the first glance. Not mentioning the humbleness and openness in sharing his knowledge during early design meetings.

I am not sure what protocol is actually used now. But I would be surprised it's different, given the protocol's simplicity and performance.

[1] https://www.cs.cornell.edu/home/rvr/papers/OSDI04.pdf [2] https://www.linkedin.com/in/allan-vermeulen-58835b/

[ryanworl]

Can you explain why the sequence numbers are so giant? I've never understood that.

[justicezyx]

I dont remember the size, is it 128bits?

It was chosen for future expansion. Kinesis was envisioned to be a much larger-scale Kafka + Storm (storm was the streaming programming framework popular in 2012, it was since falls out of favor).

[ryanworl]

128bit might be accurate, I meant more along the lines of they are non-contiguous and don't seem to be correlated with the amount of records actually being written to a stream.

[crgwbr]

I always assumed they were sequential per-region rather than per-stream, but that’s just a guess.

[justicezyx]

I recall it was sequential per-shard, but no ordering among the shards of the same stream. But I literally haven't touch Kinesis after 2013.