[prayze]

I've always been curious about this. What's the best practice for loading a large JSON file for large sets of search results? I believe when working with lunr in the past, I ended up making large network requests to load the entire JSON file at once. What's the proper way to deal with this?

[wereHamster]

Once your website reaches a certain size, the JSON will be too big to load. Then you'll have to offload the search request to a server. Either self-hosted, or a service like Algolia.

[bigiain]

You can probably push that "certain size" a long way down the road if you need to. (at least in this specific client side text search case, rather than a generic "how do I server large json files" way)

If you tweak the search box so it doesn't do anything until you've typed at least 2 or 3 letters, you could then serve regenerated son files that only contain that matches that have that prefix... No need for any of the json payload for words/phrases starting with aa..rt if someone's typed "ru" into the search box.

That means you'd have 676 distinct json files, but you'd only ever load the one you need...

[thallada]

But that requires a network request after typing to get results, which is about the same user experience of a search bar that requests some search API.

[KajMagnus]

> requires a network request

Seems to me that often, though not always, this network request would happen whilst the user is typing — say, busy typing chars 3, 4 and 5. That the netw req won't be noticeable for the human

And, if typing more chars, or backspace-deleting to fix a typo ... no netw request required.

And the same scenario, if typing a 2nd word.

I'm guessing in like 90% of the cases, it'll seem as if there was never a netw req.

[bigiain]

True, but it'd still allow your site to be a collection of static files instead of needing a executable running on a backend somewhere.

[pmlnr]

Push the corpus into SQLite, it has built-in FTS engines[^1]. Then serve it with anything. Unfortunately this needs server side code, but like 30 lines of PHP.

[^1]: https://www.sqlite.org/fts5.html

[ComputerGuru]

You can do SQLite in the browser but it’ll have to download the entire dB file instead of only opening the pages it needs (because the naive JS port can’t convert page requests to range requests).

[woadwarrior01]

It should be possible to support loading only the required pages on the browser with SQLite compiled to WASM along with a custom VFS implementation. Here’s a project[1] which does something similar (selectively load the SQLite DB on demand), albeit with the SQLite file in a torrent.

[1]: https://github.com/bittorrent/sqltorrent

[cluoma]

Wow this is great, thank you for the tip! My site uses SQLite3 for storage so this is perfect.

I'm continually amazed at how featureful SQLite is.

[sandreas]

Did anyone try to use IndexedDB instead of a big load of json with a separated json approach? I mean a bunch of static json files, that are only loaded if not already in indexeddb... like patches?

https://gist.github.com/inexorabletash/a279f03ab5610817c0540...

[nine_k]

JSON may be a very bandwidth-inefficient format. A format that can be parsed from a stream could save RAM and bandwidth, especially on mobile which is most constrained.

[mirekrusin]

However it compresses very well on the wire. One of the simplest "streamed-json" solutions is to have one json per line of input and parse that incrementally.

[_bxg1]

One thing you can do is make the data itself more concise:

- Stripping all whitespace from formatted JSON can make a huge difference

- Making property names shorter (these get repeated for every element in a large dataset!)

- If your data is relatively flat, you could replace an array of objects with an array of arrays

Or you could go all the way and serve data in CSV format, which is very space-efficient and has the neat property of being tolerant of being broken up into pieces. Though it may not parse as quickly, since JSON has native parsing support from the browser.

[speedgoose]

JSON can be compressed very well and while your advices are very good, once you compress it's not very important anymore.

[mirekrusin]

Sometimes you can partition large sets into multiple ones. If you add this extra level of indirection, you don't need to retrieve all data, just the data that is being used, which should be small fraction for some problems that work well with it. But many problems can't be solved by this approach.

[brainless]

I guess you can always load only the index in the browser, isn't it?