Hacker News new | ask | show | jobs
by iAkashPaul 813 days ago
Even notion has a similar approach to sharding postgres but both of them could benefit from having shard IDs prefixed with YY/MM/DD(as needed) otherwise it's back to the shard navigator once they max out against org-ids for each shard's capacity
1 comments
jitl 812 days ago
(I work at Notion)

Our shard key - Workspace ID - is a UUIDv4 so there’s a pretty high number of orgs per shard without conflict.

link
iAkashPaul 812 days ago
Hey Jake, I meant capacity per shard in this case, not exhausting the IDs. Any potential solutions for that or is that not an immediate challenge?
link
jitl 812 days ago
Gotcha. With our shard strategy we add more capacity either by scaling up nodes (very easy), or by resharding - adding nodes to the cluster and putting fewer shards on each node.

We recently did a reshard from 32 nodes / 15 shards per node -> 96 nodes / 5 shards per node. That puts us in footing to scale up for a while before we need to reshard again. This is a pretty smooth process, and each time we scale out we get much more scale up capacity.

Our shard logic is very simple static assignment based only on the Workspace ID. If we wanted to add workspace created time routing, we'd need to starting plumbing that information around the system in ways that are slightly annoying. Probably the move would be to re-key the Workspace table to use a date-embedding UUID format.

https://www.notion.so/blog/the-great-re-shard

link