[sgarland]

This is the only statement I disagree with:

> PowerShell gets a lot right with structured data.

CLI programs should operate on text. If you want to parse and format it, do so, but the default output mode should be plain text, so that I can pipe it into grep or awk without a second thought.

I am continuously irritated that the AWS CLI defaults to outputting in JSON. No one (I hope…) is using that tool in programs; that’s what boto3 and its ilk are for. But if humans are reading it, why default to something that they’re almost certainly going to be piping into jq if only for the formatting help?

[jcgl]

I think you're mistaking text-with-structured data for structured data itself.

Because unix shell is irrevocably text-oriented, kludging in something like JSON is basically the best that can be done when you start to want to do structured operations on structured data. (I'm sympathetic to your point about the AWS CLI tools doing JSON by default though--that just sounds like bad design.)

Being text-oriented imposes drastic limits on composability. Because there is no structure, every element of a pipeline needs to do its own parsing of the input data. This leads to brittle pipelines where every element is tightly coupled to its input's textual representation.

As an exercise, try to write a pipeline that sorts podman images by size without removing the column headers[0]:

  $ podman image ls --all 
  REPOSITORY                                 TAG         IMAGE ID      CREATED       SIZE
  docker.io/prom/prometheus                  latest      937690d77350  2 months ago  367 MB
  quay.io/keycloak/keycloak                  latest      da9433c9fac3  2 months ago  466 MB
  registry.fedoraproject.org/fedora-toolbox  43          a32da54355ca  4 months ago  2.19 GB
  docker.io/powerdns/pdns-auth-49            latest      8c1385c9deed  4 months ago  208 MB
  docker.io/testcontainers/ryuk              0.13.0      b75bc7ce94c3  6 months ago  7.21 MB

As far as I can tell, there is no way to do this in a manner that's even remotely composable. Your best bet is to basically do everything from within awk. Whatever the result would be, it certainly won't be pretty!

Contrast that with what you can do in PowerShell. You can write a couple of standalone functions[0] that are readable and composable, resulting in this pipeline:

  podman image ls --all |
      Replace-SpacesWithTabs |
      ConvertFrom-Csv -Delimiter "`t" |
      Sort-Object -Property {Convert-HumanSizeToBytes -Size $_.size} -Descending

[0] Repurposing this from a blog post I wrote: https://www.cgl.sh/blog/posts/sh.html#this-should-be-basic

[alloyed]

the point is well-taken, but i do want to show the bash version just for fun:

    podman image ls --all | sed 's/\s\s\+/\t/g' | tee >(head -n 1) >(tail -n +2 | sort -hrk 5) >/dev/null

this is _still_ all text, and we're relying heavily on sort to do a bunch of internal parsing and be in agreement with podman about how sizes should be formatted. also, for "real world" work, i dunno if the tee trick here has any kind of order guarantees, just that it works fine in this case. I'd probably just end up dropping the header and living with worse output in reality

[jcgl]

Ooh, I was so hoping someone would take up the challenge! This is a far shorter answer than I had honestly thought was possible. More readable too, somehow? Great use of tee that I would never have come up with (though I hear what you say about there maybe not being ordering guarantees).

Unfortunately, it's not 100% correct, due to misaligned headers:

  REPOSITORY TAG IMAGE ID CREATED SIZE
  registry.fedoraproject.org/fedora-toolbox 44 5a36f433c691 2 months ago 2.14 GB
  quay.io/keycloak/keycloak latest 1361d6e49205 9 days ago 478 MB
  ...

I think that speaks to your final point, which is spot-on:

> I'd probably just end up dropping the header and living with worse output in reality

This pretty much sums up plain text and unix shell imo. It's very much the pragmatic solution here, and it's what ~100% of shell scripters would choose to do. And it should make anyone question the orthodoxy around the "power" of plain text in shells.

[i15e]

A variant:

  docker image ls -a | stdbuf -oL sed -r 's/\s{2,}/\t/g' | { head -n1; tail -n+2 | sort -hrk5 -t$'\t'; } | column -ts$'\t'

I used docker since that's what I have installed and I assume the output is equivalent.

sort's -t is set to tab for field separation.

stdbuf sets sed's output to only buffer a line at a time and flush, so the head in the {...} command group doesn't completely consume stdin's contents before it's passed to tail.

The column command recreates the space-aligned table based on tab-delimited input.

[jcgl]

There is some cool stuff here.

I like using column to format the table. Appending it to alloyed's command fixes their header problem.

The stdbuf to multi-command block (term.?) is a neat trick. Although, one time when I ran this, I only got a couple lines of output. No idea why and I can't replicate it, but there could be some flakiness that results from the buffering somehow?

Question: how do the ? markers on the sort and column invocations work/what do they do?

[i15e]

I'm not seeing any ? marks in my code. Are you referring to the instances of $'\t'? In bash (and other shells, including recent versions of the POSIX sh specification) $'...' is treated like a C-style string complete with backslash escape sequences, so $'\t' is a way to have a tab character as a part of a command argument.

[jcgl]

Oops, yes, I don't know why I mistyped a question mark. That's exactly what I was wondering, thanks.

[fragmede]

It seems the sticking point is the column headers but if you as a human take them off programmatically and put them back on mentally, it's a much easier problem.

[jcgl]

That's definitely the pragmatic choice when working with shell and what roughly everyone does. But it's is also a UX compromise that, if needed anywhere other than the Stockholm Syndrome-ridden world of unix, would be routinely derided.

Shell and plain text are too embedded in the unix-descendant world to get rid of, and I'm not advocating for anything like that. Just trying to push back against oft-repeated maxims about the power of plain text. Structured data has lots to offer for composability, power, and UX.

[unfunco]

    $ podman image ls --all --sort=size

…or was the point more about doing it in a pipeline?

[jcgl]

Yes, the point was about doing it in a pipeline. The pipeline is the basis for composition of plain text in the unix shell. If something as basic as sorting a table is hard to do, it should make us question just how good the unix shell/plain text philosophy actually is.

Baking --sort flags into shell tools is a sign that the tools do not compose well.

[unfunco]

I've spent some time thinking about it, and I respectfully disagree.

I don't think a `--sort` flag is a failure of composability, the producer understands image size semantics better than the sort command, you can either bake everything into sort, making it an all encompassing command or you can simplify the previous step with a sort flag.

Your original powershell command is way more verbose and complicated, and maybe that's the price to pay for composability, but I'd rather just use the `--sort` flag and not spend hours or days coming up with some other undocumented and undiscovered way of doing it, the answer is right there ready to be used.

I think my point is that it's a violation of the UNIX principle of doing one thing really well, but I don't think that is a violation of composability.

[jcgl]

Tl;dr: Plain text's bad composability forces the dichotomy that you identify between --sort and sort. I agree with you that --sort often can be a sensible UX choice regardless, but losing out on composable middle ground is strictly a loss in terms of power and expressiveness.

Thanks for taking some time to think about it. Despite my pretty absolute wording in GP, I do think that there's nuance here. But what I want to drive home is that the tight coupling/brittleness inherent in plain text composition systematically limits composition.

What I'm not saying is that every --sort option is bad sign for composition. Like you point at, sometimes it's just a simpler UX to have such an option included with your command. As a matter of fact, you see that in PowerShell sometimes with the -Filter option on various cmdlets.

Plain text's brittleness limits composition by promoting exactly the extremes that you point at (the extremes being overloaded sort, or overloaded producer):

> the producer understands image size semantics better than the sort command, you can either bake everything into sort, making it an all encompassing command or you can simplify the previous step with a sort flag.

I think it should be obvious that baking everything into sort is a bad idea in the general case. Like you say, the producer understands the size semantics better. Moreover, the ergonomics are lousy (see comments by users alloyed and i15e).

Baking everything into --sort is bad because it limits sorting to the producer's own predefined semantics. While arguably better than relying on sort's, the producer won't always have the semantics that the user cares about. E.g. maybe the user wants to analyze the disk space used only on a certain datastore. Maybe some datastores do transparent compression and sorting should be done by physical disk usage. And so on.

These two extremes are basically your only options in a plain text world, but structured data gives you more opportunities for composition. By moving to structured data and eliminating the need for ad-hoc parsing, users and their code can operate at a higher semantic level. In particular, the loose coupling introduced by this approach gives you access to things like lambdas. You're not alone in objecting to PowerShell's verbosity. Here's a terser version that would work if podman image output structured data rather than text, thereby kinda steelmanning this position:

  podman image ls --all |
      sort {podman size -h $_.size} -d

Basically all that has happened is that the plain text->structured parsing was dropped (again, to steelman the structured data vision) and naming conventions were made unix-y.

An alternate version if no specialized podman size command is needed and the sort cmdlet would by default look at an object's size property:

  podman image ls --all |
      sort -d

In many common cases, I think I agree with you that having --size on the producer is pragmatic and fast. A good UX choice. What's bad about plain text is that there is not/cannot be any middle ground between --sort and sort.

Now, tools shouldn't always oblige the user to compose. Composition is frequently not the best UX. But a shell that systematically limits composition is prima facie worse than one that promotes it. And plain text shells do indeed limit composition.

Sorry for the essay.

[nixon_why69]

I always hated powershell for the same reason, and then I read a really long retro by the guy who ran the project and it makes sense now.

Basically Unix has a long tradition of "everything is a file" and a big ecosystem of coreutils that are based around text and windows.. didn't. You can't look at /dev or /etc and learn anything about the machine. They had a few generations of APIs and wanted to give admins and power users any shell at all instead of a GUI. So the shell is centered around making those APIs accessible, rather than piping grep and sed or whatever.

[digdugdirk]

Ooo... Any chance you remember where you read that? I'd love a reference to check it out. Sounds interesting!

[rellem]

I'm pretty sure they're referring to this one: https://corecursive.com/building-powershell-with-jeffrey-sno...

[justinhj]

Thanks for sharing that. It's relevant but also a great story about how to struggle against giant corporation bureaucracy and ship against all odds

[nixon_why69]

That's the one!

[tredre3]

Powershell commands automatically format the data for you, you can pipe it to grep just fine (I do it all the time). It's just that you can also access it in a structured way if you need to. It's the best of both worlds.

Linux tools that are starting to output raw JSON by default are indeed a nuisance, but how else can you achieve structured output if no standard shell supports it? It's a chicken and egg problem.

[em-bee]

and new shells are developing features to handle structured data like json:

here is an elvish shell command that converts a freetube playlist from json into a list of urls grouped by author:

    for i (cat 'freetube-playlist-favorites.db' | from-json)["videos"] { 
      mkdir -p $i['author']
      print http://youtu.be/$i['videoId'] >> $i['author']/get }

here is one to get a list of devices connected to my zerotier network

    curl -s -H "Authorization: token <redacted>" "https://api.zerotier.com/api/v1/network/<redacted>/member" | 
      all (from-json) | 
      order &key={|d| put $d[name]} | 
      each { |device| 
             var t = (printf "%.0f" (/ $device[lastSeen] 1000))
             if (> 20000000 (- (date '+%s') $t))  { 
               print (date -R --date='@'$t) $device[config][ipAssignments] $device[name] "\n" } }

those are not scripts saved in a file. i run these directly on the commandline. ignore the elvish syntax, focus on the ease of accessing values from the json data. those are just two examples, though i recently discovered an ls replacement that optionally outputs json, that will be interesting to use.

[jcgl]

Great example! How do you like using elvish? Even though I am a proponent of structured data and like PowerShell a lot (mentioned in a nearby comment of mine), I use fish as my regular shell. Big fan of fish's careful focus on user experience, but would be open to trying something structured.

[blooalien]

I'ma huge fan of `fish` myself, but I've been watching `elvish` for a while now, and some of their scripting stuff really is awfully nice.

[fragmede]

Interesting. That's a lot to type on the fly, but I can posit some benefits.

[zbentley]

Slightly off topic, but it is worth being familiar with AWSCLI’s built in jmespath JSON transformation system. That saves on the need for jq in a lot of cases. Given that awscli bundles that, I generally forgive it for defaulting to JSON (especially given how deeply nested and interlinked many AWS API results are), but reasonable minds can differ. AWSCLI’s schizophrenic use (or not) of pagers for tiny results, on the other hand, is less defensible.

[hombre_fatal]

I'm not sure I agree.

In the AWS case the tools talk to an API server, so sure, you can call the API server directly or use a wrapper that does, but what about all the other CLI apps that don't? The CLI program is the API.

I built a CLI program that wraps luks + btrfs, and they only offer a `--json` output option for a few commands. I have to write an ad hoc parser for each command since raw text includes arbitrary formatting and presentation lipstick that the creator came up with. And I have to do extra work to avoid breaking changes at the parser level instead of the higher data level.

If I had to pick between the two, json would at least solve the data representation part so that I can build on top of it. And it's trivial to go data (json) -> pretty print rather than pretty print -> data.

I can see it being annoying if all you care about is using CLI programs by hand, but it seems like a mild upside compared to the downsides the second you want to consume it programmatically, even if it's with a chain of awk, cut, and tr.

[tialaramex]

I thought I would like Powershell but it was another big disappointment for me.

It's got a lot of the unpleasant clunkiness that something like the Bourne shell owes to decades of compatibility, but Microsoft doesn't have that excuse. Despite this, it's gratuitously incompatible with itself, I had code which worked fine, we upgraded Powershell oh, now that won't work, just fix all the scripts. Crazy. It clearly wants an NPM-style experience where you seamlessly incorporate other people's work, but then it doesn't really deliver this well and so you often end up manually copy-pasting.

If Powershell was a one man project and this was their Beta I'd say it is promising. But it's a project from the Microsoft corporation for 20 years. Do better.

[torginus]

This is my opinion as well. Powershell and object shells are much better and the way to go imo. Also Powershell code is fast owing to running on the .NET VM.

But the actual UX is just convoluted and horrible. For example, the person who decided that commands without a ; at the end of the line should just vomit their outputs to the enclosing function's clearly hasn't heard of the principle of least surprise.

PS just does so many plain weird and unintuitive things, which is worsened by the fact that it looks like a boring old programming language, while clearly not being one.