[theamk]

(1) Try a language with fast allocations (C, C++, Rust, maybe Go or Java) -- anything except Python or Ruby

or

(2) Try using streaming API (I don't know Ruby, but quick google found https://github.com/dgraham/json-stream ). Note that this method will require you to massively restructure your program -- you want to avoid having all of the data in memory at once.

The streaming API might work better with jq-based preprocessing -- for example, if you want to compare two unsorted sets, it may be faster to sort them using jq, then compare line-by-line using streaming API.

[dith3r]

Python is fast at parsing JSON, Go had hard time to match parsing speed of it. Additionally you have PyPy to help.

[mattbillenstein]

Nod. https://gist.github.com/mattbillenstein/34cf2907390102ffbabd...

[Filligree]

Python is fast at doing anything that doesn't involve running Python.

That's an important caveat. Python's C JSON parser library is super-fast, but if you want to use the data for anything but a simple equality check afterwards, it'll be slow as molasses.

Or you'll write a C extension for it...