Hacker News new | ask | show | jobs
by yeldarb 2675 days ago
Will this work on JSON files that are larger than the available system memory?

Firebase backups are huge JSON files and we haven’t found a good way to deal with them.

There are some “streaming JSON parsers” that we have wrestled with but they are buggy.

2 comments

Sadly it will not. Arguably we could 'stream' things, but we don't have an API or a use case for it. If you could capture your requirements and put them on an issue on Github, it would be helpful. We're not against the streaming use case, we just don't understand it very well.
Probably not. I requires a memory allocation the size of the file for parsing.

However they have the ability to build a tape out of the json and find the interesting marks. Perhaps it can be adapted to make a fast parser than only parses the relevant stuff but zooms through the large file in blocks.