[tpmoney]

For fun, I took a crack at your example and came up with this craziness (with the caveat it's late and I didn't spend much time on it), which is made a bit more awkward because grep doesn't do capturing groups:

  netstat -aln \
  | grep ESTABLISHED \
  | awk '{print $4}' \
  | grep -Po '\:\d+$' \
  | grep -Po '\d+' \
  | sort \
  | uniq -c \
  | sort -r \
  | head -n 10

Changing the awk field to 5 instead of 4 should get you remote ports instead of local. But yeah, that will be fragile if netstat's output ever changes. That said, even if you're piping objects around, if the output of the thing putting out objects changes, your tool is always at risk of breaking. Yes objects breaking because field order changed is less likely, but what happens if `Get-NetTCPConnection` stops including a `State` field? I guess `Where-Object` might validate it found such a field, but I could also see it reasonably silently ignoring input that doesn't have the field. Depends on whether it defaults to strict or lenient parsing behaviors.

[jiggawatts]

I know this sounds like nit-picking but bear with me. It's the point I'm trying to make:

1. Your script outputs an error when run, because 'bash' itself doesn't have netstat as a built-in. That's an external command. In my WSL2, I had to install it. You can't declaratively require this up-front, you script has to have an explicit check... or it'll just fail half-way through. Or do nothing. Or who knows!?

PowerShell has up-front required prerequisites that you can declare: https://learn.microsoft.com/en-us/powershell/module/microsof...

Not that that's needed, because Get-NetTcpConnection is a built-in command.

3. Your script is very bravely trying to parse output that includes many different protocols, including: tcp, tcp6, udp, udp6, and unix domain sockets. I'm seeing random junk like 'ACC' turn up after the first awk step.

4. Speaking of which, the task was to get tcp connections, not udp, but I'll let this one slide because it's an easy fix.

5. Now imagine putting your script side-by-side with the PowerShell script, and giving it to people to read.

What are the chances that some random person could figure out what each one does?

Would they be able to modify the functionality successfully?

Note that you had to use 'awk', which is a parser, and then three uses of 'grep' -- a regular expression language, which is also a kind of parsing.

The PowerShell version has no parsing at all. That's why it's just 4 pipeline expressions instead of 9 in your bash example.

Literally in every discussion about PowerShell there's some Linux person who's only ever used bash complaining that PS syntax is "weird" or "hard to read". What are they talking about!? It's half the complexity for the same functionality, reads like English, and doesn't need write-only hieroglyphics for parameters.

[tpmoney]

Because I didn't see the edited version when I was writing my original reply and its too late, I want to call out another problem that you graciously overlooked that we can call #2 since it touches neatly on your #1 and #3 items and #2 is already missing. The extra junk you see in your wsl after the awk step is probably because the other big *NIX problem with shell scripts is my `netstat` or `grep` or even `echo` might not be the same as yours. I originally wrote it on a mac, and while I was checking the man page for netstat to see how old it was and how likely netstat output would change, it occurred to me that BSD netstat and linux netstat are probably different, so I jumped over and re-wrote on a linux box. Entirely possible your version is different from mine.

Heck, just checking between Bash, ZSH and Fish on my local machine here and Bash and ZSH's version is from 2003 and provides a single `-n` option and declares POSIX compliance, but explicitly calls out that `sh`'s version doesn't accept the `-n` argument. Fish provides their own implementation that accepts arguments `[nsEe]` from 2023. Every day I consider it a miracle that most of the wider internet and linux/unix world that underlies so much of it works at all, let alone reliably enough to have multiple nines of uptime. "Worse is better" writ large I guess.

[jiggawatts]

I was worried that my toy problem wasn’t complex enough to reveal these issues!

I had an experience recently trying to deploy an agent on a dozen different Linux distros.

I had the lightbulb moment that the only way to run IT in an org using exactly one distro. Ideally one version, two at the most during transitions. Linux is a kernel, not an operating system. There are many Linux operating systems that are only superficially “the same”.

[sgarland]

At least here, we can agree. If I ran a business and allowed employees to run Linux (which is reasonable, IMO), the last thing I want is someone's riced-out Gentoo with an unpatched security exploit onto the VPN.

[tpmoney]

Sure, I'm not arguing that having a set of well defined outputs and passing objects around wouldn't be better. You're talking to someone that often laments that SmallTalk was not more popular. But you'd need to get the entire OSS community to land on a single object representation and then get them to independently change all the tools to start outputting the object version. PowerShell and Microsoft have the advantage in this case of being able to dictate that outcome. In the linux world, dictating outcomes tends to get you systemd levels of controversy and anger.

Technically speaking though, there's no reason you couldn't do that all in bash. It's not the shell that's the problem here (at least, to an extent, passing via text I guess is partly a shell problem). There's no reason you couldn't have an application objNetStat that exported JSON objects and another app that filtered those objects, and another that could group them and another that could sort them. Realistically "Sort-Object Count -Descending -Top 10" could be a fancy alias for "sort | uniq -c | sort -r | head -n 10". And if we're not counting flags and arguments to a function as complexity, if we had our hypothetical objNetstat, I can do the whole thing in one step:

  objNetstat --json \
  | jq -c '[.[] | select(.state == "ESTABLISHED")] | group_by(.port) | map({port:.[0].port, count: map(.host) | length}) | sort_by(.count) | reverse | [limit(10;.[])]'

One step, single parsing to read in the json. Obviously I'm being a little ridiculous here, and I'm not entirely sure jq's DSL is better than a long shell pipe. But the point is that linux and linux shells could do this if anyone cared enough to write it, and some shells like fish have taken some baby steps towards making shells take advantage of modern compute. Why RedHat or one of the many BSDs hasn't is anyone's guess. My two big bets on it are the aversion to "monolithic" tools (see also systemd), and ironically not breaking old scripts / current systems. The fish shell is great, and I've built a couple shell scripts in it for my own use, and I can't share them with my co-workers who are on Bash/ZSH because fish scripts aren't Bash compatible. Likewise I have to translate anyone's bash scripts into fish if I want to take advantage of any fish features. So even though fish might be better, I'm not going to convince all my co-workers to jump over at once, and without critical mass, we'll be stuck with bash pipelines and python scripts for anything complex.

[jiggawatts]

Half way through your second paragraph I just knew you'd be reaching for 'jq'!

All joking aside, that's not a bad solution to the underlying problem. Fundamentally, unstructured data in shell pipelines is much of the issue, and JSON can be used to provide that structure. I'm seeing more and more tools emit or accept JSON. If one can pinch their nose and ignore the performance overhead of repeatedly generating and parsing JSON, it's a workable solution.

Years ago, a project idea I was really interested for a while was to try to write a shell in Rust that works more like PowerShell.

Where I got stuck was the fundamentals: PowerShell heavily leans on the managed virtual machine and the shared memory space and typed objects that enables.

Languages like C, C++, and Rust don't really have direct equivalents of this and would have to emulate it, quite literally. At that point you have none of the benefits of Rust and all of the downsides. May as well just use pwsh and be done with it!

Since then I've noticed JSON filling this role of "object exchange" between distinct processes that may not even be written in the same programming language.

I feel like this is going to be a bit like UTF-8 in Linux. Back in the early 2000s, Windows had proper Unicode support with UTF-16, and Linux had only codepages on top of ASCII. Instead of catching up by changing over to UTF-16, Linux adopted UTF-8 which in some ways gave it better Unicode support than Windows. I suspect JSON in the shell will be the same. Eventually there will be a Linux shell where everything is always JSON and it will work just like PowerShell, except it'll support multiple processes in multiple languages and hence leapfrog Windows.

[tpmoney]

>Years ago, a project idea I was really interested for a while was to try to write a shell in Rust that works more like PowerShell.

So this whole conversation and a different one about python and it's behavior around `exit` vs `exit()` sent me down a rabbit hole of seeing if I could make the python interpreter have a "shell like" dsl for piping around data. It turns out you sort of can. I don't think you can defeat the REPL and make a function call like `echo "foo" "bar" "baz", but you can make it do this:

  netstat("-ln") | filter_by({"state":"ESTABLISHED"}) \
    | group_by(["local_port"]) | count_groups \
    | sort("count", descending=True) | limit(10)

And only ever parse plain text once on the input from netstat. For your amusement/horror I present "ShPy": https://gitlab.com/tpmoney/shpy

[tpmoney]

>Years ago, a project idea I was really interested for a while was to try to write a shell in Rust that works more like PowerShell. >Where I got stuck was the fundamentals: PowerShell heavily leans on the managed virtual machine and the shared memory space and typed objects that enables.

Hmmm, if you need that sort of shared memory access throughout the shell, you probably need a language like python (or maybe better Lisp) with a REPL and the ability/intent to self modify while running. Of course, every time you have to farm out because you don't have a re-written replacement internally to the shell app, you'd still parsing strings, but at least you could write a huge part of data processing in the shell language and keep it in house. Years ago I worked for a company that was using Microsoft's TFVC services (before it was azure devops or whatever they call it now) and wrote a frontend to their REST API in python that we could call from various other scripts and not be parsing JSON everywhere. Where this is relevant to the discussion is that one of the things I built in (in part to help with debugging when things went sideways) was an ability to drop into the python REPL mid-program run to poke around at objects and modify them or the various REST calls at will. With well defined functions and well defined objects , the interactive mode was effectively a shell for TFVC and the things we were using.

Though all of that said, even if one did that, they would still either need to solve the "object model" problem for disparate linux tools, or worse commit to writing (or convincing other people to write and maintain) versions of all sorts of various tools in the chosen language to replace the ones the shell isn't farming out to anymore. Its one thing to chose to write a shell, it's something else entirely to choose to re-write the gnu userland tools (and add tools too)

[samatman]

> Years ago, a project idea I was really interested for a while was to try to write a shell in Rust that works more like PowerShell.

Today's your lucky day!

https://www.nushell.sh

[sgarland]

> PowerShell has up-front required prerequisites that you can declare

Anyone who's written more than a few scripts for others will have learned to do something like this at the start:

    declare -a reqs
    reqs+=(foo bar baz)
    missing=0
    for r in "${reqs[@]}"; do
        if (! command -v "$r" &>/dev/null); then
            echo "${r} is required, please install it"
            missing=1
        fi
    done
    if [ $missing -gt 0 ]; then
        exit 1
    fi

> Your script is very bravely trying to parse output that includes many different protocols, including: tcp, tcp6, udp, udp6, and unix domain sockets

They probably didn't know you could specify a type. Mine only displays TCP4.

> Now imagine putting your script side-by-side with the PowerShell script, and giving it to people to read.

I'm gonna gatekeep here. If you don't know what that script would do, you have no business administering Linux for pay. I'm not saying that in a "GTFO noob" way, but in a "maybe you should know how to use your job's tools before people depend on you to do so." None of that script is using exotic syntax.

> Note that you had to use 'awk', which is a parser, and then three uses of 'grep' -- a regular expression language, which is also a kind of parsing.

They _chose_ to. You _can_ do it all with awk (see my example in a separate post).

> Literally in every discussion about PowerShell there's some Linux person who's only ever used bash complaining that PS syntax is "weird" or "hard to read". What are they talking about!? It's half the complexity for the same functionality, reads like English, and doesn't need write-only hieroglyphics for parameters.

And yet somehow, bash and its kin continue to absolutely dominate usage.

There is a reason that tools like ripgrep [0] are beloved and readily accepted: they don't require much in the way of learning new syntax; they just do the same job, but faster. You can load your local machine up with all kinds of newer, friendlier tools like fd [1], fzf[2], etc. – I definitely love fzf to death. But you'd better know how to get along without them, because when you're ssh'd onto a server, or god forbid, exec'd into a container built with who-knows-what, you won't have them.

Actually, that last point sparked a memory: what do you do when you're trying to debug a container and it doesn't have things like `ps` available? You iterate through the `/proc` filesystem, because _everything is a file._ THAT is why the *nix way exists, is wonderful, and is unlikely to ever change. There is always a way to get the information you need, even if it's more painful.

[0]: https://github.com/BurntSushi/ripgrep

[1]: https://github.com/sharkdp/fd

[2]: https://github.com/junegunn/fzf

[gjvc]

put the pipe character at the end of the line and you don't need the backslashes