Hacker News new | ask | show | jobs
by juancn 1998 days ago
The hardest part is to be able to parallelize the sorts in the backend and keep the working sets reasonably sized.

If you ask for "first 50 items after key X", you just need to keep a priority queue of size 50 on each BE node and merge them before returning them (I'm assuming a distributed backend). It doesn't matter on which page you are.

But if you specify "first 50 items after element N" it gets really tricky, each BE shard needs to sort the first N elements, and it can use some trickery to avoid doing a naive merge (see: https://engineering.medallia.com/blog/posts/sorting-and-pagi... ).

You can at most save some transfer over the network.