Hacker News new | ask | show | jobs
by citrin_ru 15 days ago
> The example shows an auto-incrementing user ID. How’s that possible without contention between all shards?

Why not divide ID range (63bit?) by the maximum planned number of shards and then set on each shard it's min/max value so ranges will not overlap?
1 comments
chiph 15 days ago
Because if the business grows, then you will need to add servers, and reallocate each existing shard's min/max values. If you left room to grow (divide amongst 2^31 instead of 2^63) you can go a long way before you have to solve that problem. If the business turns out to be wildly successful, you'd probably set up a whole new pool anyway and migrate over (and get a new set of pre-allocated ID ranges).

This might be the perfect case to use an UUIDv7 as a key instead of a number.

link
prirun 15 days ago
Consistent hashing helps solve this. You don't divide into shards based on the id, but on the consistent hash of the id. Then when you have to add shards, it's easier to distribute data to the new shards. If you have 3 shards and expand to 4, only 1/4th of the data from each shard gets moved to the new shard.
link