| HN Mirror

Hmm. I also think jq is more awkward than it needs to be, but I don’t think the points you mention are a problem. Maybe explaining them would help?

(Note: the following explains jq’s operation using the smallest possible subset of the language, it doesn’t aim to use the most natural programs possible.)

So jq’s data model (much like XPath’s actually) is that everything is a (possibly empty) stream of (JSON) values. On input (unless you use -s), it accepts any number of concatenated JSON objects (usually separated by newlines or ACSII RS, but as JSON is self-delimiting that isn’t strictly required) and makes a stream out of them.

That is then fed into the program, a pipeline of |-separated transforms, each of which can generate zero or more output elements from each input element. For example, .foo is a one-to-one transform that, when it accepts an object, emits the value of its foo property (and fails otherwise):

  $ echo '{"foo": null}{"bar": 1, "foo": 42}' | jq .foo
  null
  42

And .[] is a one-to-zero-or-more transform that, on accepting an array, emits each array element separately (and fails otherwise):

  $ echo '[false,1][][2]' | jq '.[]'
  false
  1
  2

While select(F) is a one-to-zero-or-one transform that, on accepting an element, feeds it into F and lets its pass through if it got a truthy value or rejects if it got a falsy one:

  $ echo '{"foo": null}{"bar": 1, "foo": 42}' | jq select(.foo)
  {"bar": 1, "foo": 42}

OK, that last one was a bit of a lie. Because we don’t want to introduce functions into the language as a separate kind of thing to transforms, F is also a transform, so it might possibly emit more than one value in response to whatever select fed it. The full truth is that select(F) is a one-to-zero-or-more transform that emits each input value as many times as there are truthy values in F’s response to it:

  $ echo 'false 42' | jq 'select([true, "also truthy"] | .[])'
  false
  false
  42
  42

That might have not been terribly useful, but it illustrates two points. First, a JSON literal is a valid transform: one that emits itself every time it gets something. (That’s why you need to write .[] for flatten: plain [] is the empty array literal.) Second, while jq cannot do many-to-one transforms, on pain of losing its streaming nature, it can do something like nested contexts, where it launches a subordinate pipeline and does something with the results.

And it is willing to collect those results instead of streaming them: if you have a pipeline P, [P] is a one-to-one transform that, for each input element, runs P on it, collects all the results from them, puts them into an array and emits that. For example:

  $ echo '[[0,1],[2]] [[]] [] [[3]]' | jq '[.[] | .[]]'
  [0,1,2]
  []
  []
  [3]

Or:

  $ echo '[false,1][][2]' | jq '[.[] | select(.)]'
  [1]
  []
  [2]

(here . is the one-to-one identity transform). Instead of [.[] | P] you can write map(P).

What this boils down to select(C) will go through the input stream and pare down its elements to those that satisfy C, while map(select(C)) will go through the input stream of arrays and pare down each array’s elements to those that satisfy C.

Final point: if you want to give up streaming, the -s / --slurp flag will slurp the input stream into an array, then feed it to your program as a single element. That is, jq -s '.[] | P' is a worse synonym of jq P.