Hacker News new | ask | show | jobs
by t0astbread 2004 days ago
Just a quick non-thought-through idea but would it be possible to build an index in a way that allows clients to download only parts of it based on what they search? I.e. the search client normalizes the query in some way and then requests only the relevant index pages. The index would probably be a lot larger on the server but if disk space is not an issue...?

(Though at some point you have to ask yourself what the benefits of such an approach are compared to a search server.)
2 comments
lgierth 2003 days ago
Merkle Search Trees: Efficient State-Based CRDTs in Open Networks https://hal.inria.fr/hal-02303490/document

Peer-to-Peer Ordered Search Indexes https://0fps.net/2020/12/19/peer-to-peer-ordered-search-inde... (which adds useful context about the above)

Sorry for just dropping these links, I should already be asleep :)

link
jil 2003 days ago
I keep wanting to try that! Sharding the index would be really cool, and would definitely allow for a seemingly infinite index size. The barrier, as you were getting to, would be making sure index shard downloads could keep up with typing speed.

Definitely something I'll try experimenting with in the future, so... watch this space, perhaps?

link