Hacker News new | ask | show | jobs
by brundolf 1405 days ago
It's very possible it could be faster; jq seems to actually be fairly unoptimized. This implementation in OCaml was featured on HN a while back and it trashes the original jq in performance: https://github.com/davesnx/query-json

After seeing that one I did my own (less-complete) version in Rust and managed to squeeze out even more performance in the operations it supports: https://github.com/brundonsmith/jqr

2 comments

Working with large json files is hard to parallelize. Just filtering the objects in a root array can take very long. jqr and gojq both die with OOM when running on large files like

https://dumps.wikimedia.org/wikidatawiki/entities/latest-all...

A fast tool to split a json file like that into a format with one json file per line would already help a lot.

Mine can :)
jq is particularly bad at large stream progressing