[lloeki]

Content negotiation.

Imagine a magical `ls` that can emit text/plain, application/json, what have you:

  ls -f text
  ls -f json
  ls -f csv
  ls -f msgpack
  ...

Now instead of specifying formats on both ends:

  ls | jq             # jq accepts application/json => ls outputs as json
  ls | fq             # fq accepts a ton of stuff => "best fit" would be picked
  ls | fq -d msgpack  # fq accepts only msgpack here => msgpack
  ls                  # stdout is tty, on the other end is the terminal who says what it accepts => human readable output

Essentially upon opening a pipe a program would be able to say what they can accept on the read end and what they can produce on the write end. If they can agree on a common in-memory binary format they can literally throw structs over the fence - even across languages, FFI style - no serialisation required, possibly zero-copy.

We know how to do that:

- https://www.rfc-editor.org/rfc/rfc2616#page-71

- https://www.rfc-editor.org/rfc/rfc2616#page-100

And I mean, we really know: the last one we already do! Tons of programs check for stdin and/or stdout being tty via an ioctl and change their processing based on that.

It'd allow a bunch of interesting stuff, like:

- `cat` would write application/octet-stream and the terminal would be aware that raw binary is being cat'd to its tty and thus ignore escape codes, while a program aiming to control the tty would declare writing application/tty or something.

- progressive enhancement: negotiation-unaware programs (our status quo) would default to text/plain (which isn't any more broken that the current thing) or some application/unix-pipe or something.

- when both ends fail to negotiate it would SIGPIPE and yell at you. same for encoding: no more oops utf8 one end, latin1 on the other.

[wwader]

Interesting post! when prototyping what would end up being fq i did quite a lot of tinkering with how it could work by splitting it up into multiple tools, and use it like: <decode tool> | <query tool> | <display tool>. Never got it to feel very neat and problem is what would be piped? i tried JSON so that <query tool> could jq but whatever is would be would have to be quite verbose to be able for <display tool> to be able to show hexdump dumps, value mappings, descriptions etc. So in the end i ended up doing more or less the same design but in jq where the values piped between filters is kind of a "polymorphic" JSON values. Those behave behave like JSON value but can be queried for which bit range and buffer they originate from or if the value is symbolic representation, description etc. Maybe an interesting note about fq is that for format like msgpack it kinds of can decode in two modes, by default it decodes msgpack into like a "meta"-mode where things like integer encoding, sizes etc can be seen and then there is another "representation"-mode that is JSON value of what it represents.

[gregjor]

That's not an improvement. Have you heard of "cat -v considered harmful?"

http://harmful.cat-v.org/cat-v/

[bigbillheck]

I think we've all heard of it, but I for one don't find it persuasive.