[garethrowlands]

Articles like this are fun but they all come from posix shell syntax being fundamentally bad for scripting/programming. All the piping stuff is great, of course. And the overall ecosystem is great. But the interpretation of the script itself working by a series of string substitutions is a mechanism we wouldn't accept in a regular programming language. And there's no excuse for it really, except that shell syntax is really, really old.

For example, what does `$foo` mean in shell syntax? In any reasonable language (perl or powershell, for example, or python if you drop the `$`), it's an expression that evaluates to whatever value's inside that variable. In shell, `$foo` isn't an expression in that sense, and what it does depends on what's inside it via a variety of string substitution rules.

This is the main reason we have arcane articles like this.

That said, nice article.

[bityard]

I feel the need to push back on this perspective.

Old doesn't imply arcane. It instead can (and does, in this case) mean that it won out over the course of decades against other less worthy alternatives.

Although you can write "programs" in sh, the shell syntax was never meant to be anything like a systems or application programming language. It was meant to efficiently automate systems tasks. It is kind of like the fork lift of computing. It does it's job very well, is bad at doing anything else, yet no other kind of vehicle can do it's job nearly as efficiently.

Comparing the shell to C, Go, Rust, Python, or JavaScript is crazypants. It has a different job than those, so of course it will look and feel different!

[garethrowlands]

There are plenty of alternative shells that don't suffer these problems. Posix syntax didn't win because it was better - it's the classic 'worse is better'.

> Old doesn't imply arcane.

Indeed so. For example, awk is about the same age but doesn't suffer the same problems.

> the shell syntax was never meant to be anything like a systems or application programming language

A programmable shell is not only like a programming language, it is a programming language. Systems tasks have some specialised requirements but, overall, scripting is programming. There is nothing in the problem domain that requires the posix string-substitution problems.

If fish or oil had existed back then, they would likely have won, since they are better.

[dahart]

> overall, scripting is programming.

This is a logical statement that is technically true while not matching real-world experience. Scripting as a type of programming is different from application development programming, and scripting needs differences in features and interface to be convenient and comfortable.

> There is nothing in the problem domain that requires the posix string-substitution problems.

What do you mean? Environment variables and command line execution both separately lead to string substitutions. The fish shell uses string substitutions.

You emphasized ‘requires’ which again might make your statement technically true while still missing the experiential forest for the logical trees. You can use x86 assembly for your shell if you want, there’s nothing that “requires” posix. The reason string substitutions exist is because they’re useful and convenient; the alternatives are verbose and klunky.

[rmwaite]

I think the point is that worse is better. There is incredible power in a standard (mostly) predictable platform, even if it could be done better.

[andrewl]

I think Bash won as Bityard says above, in the sense that COBOL won, which was because it was everywhere (approximately) and then when another system was being built to do something in COBOL’s niche COBOL was the obvious choice. Once it was sufficiently established it won by inertia and absolutely not because it was the better language. Or maybe it was the better language in its time, but it certainly is not now. And note I’m only talking about COBOL’s niche, of mainframe-based processing in the banking, insurance, state and federal payroll and similar areas.

[tyre]

In a similar way that Python won, despite having some pretty delulu characteristics.

Whitespace sensitivity is horrendous (can’t autoformat, no multi-line lambdas, bonkers inline list comprehensions) but here we are.

[cogman10]

> It was meant to efficiently automate systems tasks.

Shell was designed to launch processes easily. Unix philosophy pushed to make every function an application. And that's ultimately what shell "syntax" is.

There's shockingly little syntax in the shell. A lot of what we think of as "shell syntax" is actually full blown (standard) programs which the shell is launching. My favorite of which is `[`.

What makes shell unique vs all other programming languages is it puts launching processes first. It has no friction there because if it did, it wouldn't work.

This is also what I think is gross about shell. Every other scripting language has at least some friction when launching other processes and that's usually a good thing IMO.

[lynndotpy]

One of these are not like the other. Python is perfectly serviceable as a shell, to the extent that one could largely replace bash with the Python REPL if they wanted.

[tejtm]

python you say?

Pray tell which one of the myriad is the true one?

And do not worry, no one would be rash enough to bring in external dependencies it is just too darn inconvenient.

I should probably drink my coffee before posting, be less grumpy.

[lynndotpy]

I think you are reading something into my post that isn't there. You surely do not believe I am arguing that Python is a shell language without tradeoffs.

Of the languages listed in GP, only Python and JavaScript are scripting languages with REPLs like Bash. Of those two, Python is far more serviceable for what one uses bash for, even in embedded applications (with MicroPython).

Perl, Awk, Ruby, and Lua are also scripting languages one can use in the course of OS scripting. (Your distro probably uses some combination of these, including Python.) Of these, at least Ruby can also be used as a shell.

> Pray tell which one of the myriad is the true one?

You've basically got it, that's just `which python`

I use Python for longer scripts where script maintainability is a priority and external libraries are unnecessary, and as a general shell when I also want my shell to be a calculator, especially when computing on a smartphone (with an appropriate pythonrc).

The key thing is that I already know Python and I find Python's warts more palatable than those on Bash.

[andrehacker]

I think the point was that Python, as a language and ecosystem, has experienced a number of breaking changes over the years. In practice, that means code written only a few years ago may no longer run without specific versions of Python and a collection of dependencies, many of which have since been replaced by newer, incompatible implementations.

What I appreciate about the traditional shell languages is their remarkable stability. Shell scripts written in 2000, or even earlier, are often still able to run today with little or no modification. By contrast, Python applications frequently require recreating a historical runtime environment, including older language versions and dependencies, many of which have accumulated significant security vulnerabilities over time.

Interestingly, apart from the various shell languages, Perl is probably one of the strongest alternatives in this regard. The Perl community has placed a high value on backward compatibility, allowing older code to continue functioning while the language itself remains actively maintained and up to date.

[lynndotpy]

I basically agree with these tradeoffs (you should never be pulling in dependencies outside the builtins), but let's look back to GP:

> Comparing the shell to C, Go, Rust, Python, or JavaScript is crazypants. It has a different job than those, so of course it will look and feel different!

The point was just a small one, that Python is not like the others. C, Go, and Rust do not come with REPLs. Node is plausible to use as an OS shell but I've never heard of anyone doing that. Python is the only one of those I've used as a shell or have heard of others using as a shell.

Again, I'm not advocating for Python as a good general-purpose shell. I'm only making the small claim that C, Go, Rust, and (to a smaller extent) JavaScript cannot be be used like bash. Python could be used as a shell.

[amiga386]

I'd be jumping out the top storey window after about 2 minutes of being required to use Python as a shell.

Python is a full-on programming language. Its REPL is a REPL for that language. Interacting with the OS and filesystem is a niche, tucked away in a corner of the language.

It has has none of the ergonomic affordances needed to be the user interface for interacting with files and processes.

Here are some simple commands:

    ls
    cd foo
    ls
    rm bar
    cd ../baz
    rm bar
    echo >readme.txt baz directory
    curl -s http://example.com >download

Let's try doing that with python:

    python
    >>> ls
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'ls' is not defined
    >>> import system
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ModuleNotFoundError: No module named 'system'
    >>> import os
    >>> os.dir.list()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: module 'os' has no attribute 'dir'
    >>> help(os)

    >>> os.listdir()
    ['foo', 'bar', 'baz']
    >>> cd foo
      File "<stdin>", line 1
        cd foo
           ^
    SyntaxError: invalid syntax
    >>> chdir("foo")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'chdir' is not defined
    >>> os.chdir("foo")
    >>> os.listdir()
    ['bar']
    >>> os.delete("bar")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: module 'os' has no attribute 'delete'
    >>> os.remove("bar")
    >>> os.chdir("../baz")
    >>> os.remove("bar")
    >>> f = open("readme.txt", "w")
    >>> f.write("baz directory")
    13
    >>> f.close()
    >>> import process
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ModuleNotFoundError: No module named 'process'
    >>> import subprocess
    >>> subprocess.run("curl", "-s", "http://example.com/")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib/python3.9/subprocess.py", line 505, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/usr/lib/python3.9/subprocess.py", line 778, in __init__
        raise TypeError("bufsize must be an integer")
    TypeError: bufsize must be an integer
    >>> subprocess.run(["curl", "-s", "http://example.com/"])
    ...
    CompletedProcess(args=['curl', '-s', 'http://example.com/'], returncode=0)
    >>> subprocess.run(args=["curl", "-s", "http://example.com/"], capture_output=True)
    CompletedProcess(args=['curl', '-s', 'http://example.com/'], returncode=0, stdout=b'...\n', stderr=b'')
    >>> p = subprocess.run(args=["curl", "-s", "http://example.com/"], capture_output=True)
    >>> fh = open("download", "w")
    >>> fh.write(p.stdout)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: write() argument must be str, not bytes
    >>> fh = open("download", "wb")
    >>> fh.write(p.stdout)
    559
    >>> fh.close()

[lynndotpy]

Yes, someone who has no idea how to use their shell and tries to get around by making arbitrary guesses will suffer. This is true for bash as well. It's no reason to kill yourself immediately.

[amiga386]

I put it to you that a "shell" that does not natively run the executable programs you type is a non-starter as a shell.

I would be ready for the window if I had to type

    import subprocess; p = subprocess.run(args=["curl", "-s", "http://example.com/"], capture_output=True); fh = open("output", "wb"); fh.write(p.stdout); fh.close()

rather than

    curl -s http://example.com/ >output

There are certainly attempts to create a shell that's powered by Python and allows the Python language into it, but one could not "largely replace bash with the Python REPL". Only people who didn't want a shell in the first place would think that reasonable.

[lynndotpy]

I think we're talking past each other. You are largely not understanding the point I'm saying. I agree that Python, as a shell and scripting language, will have different ergonomics and tradeoffs compared to Bash.

The point I am making is that, of the listed languages, Python is far more comparable to Bash than the others. Suppose you had no choice but to use one of the following executables in place of /usr/bin/bash

- go - gcc - rustc - node - python

You would choose Python, of course. Plausibly node, but certainly not any of the other three.

[dahart]

> fundamentally bad for scripting/programming […] a mechanism we wouldn’t accept in a regular programming language

I don’t agree with this assumption that shell should use mechanisms we accept in “regular” programming languages, whatever that means. And why not the other way around? Why do we accept mechanisms in programming that we wouldn’t accept in a shell? There are layers of assumption in your assumption.

I also disagree with lumping scripting and programming together. Those are two different activities, and shell is better at scripting than programming, and it’s better at scripting than programming languages.

Try using python as your shell, and you’ll find out that python is fundamentally bad as a shell REPL. The shell is necessarily different from python or C++ or <your_favorite_programming_language>, starting with the ability to run command lines by typing them, with zero extra syntax. That is something regular programming languages can’t do, and it leads directly to the need for string substitution rules.

If you want a shell made out of a good programming language… make one! People have been thinking about this for decades and nobody has come up with a good one, because shell languages like bash are better at shelling than programming languages.

[chrisweekly]

a bunch of happy xonsh users disagree

[dahart]

Hey good! I’d like a good python shell. Does xonsh fix cut and paste with python? The first huge problem I ran into with python shelling is you can’t cut as paste indented code at all.

Edit I’m reading about it, here are some notes to myself:

Sounds like there are 2 modes, subprocess & python

Env vars are still string substitution, and there are many rules and string literal prefixes.

You will want the prompt toolkit (ptk). It’s an additional install. (Non-default installs, btw, are why I stopped using zsh. I loved zsh, but had to move to bash.)

Regexes use backticks

@.imp for inline imports… good idea

BTW I googled for xonsh complaints and got what I expected; awkward & klunky interactive editing, special prefix syntax, some missing features, system python dependency issues, high CPU usage, and lack of portability. People seem to suggest that xonsh is better for scripting than interactive usage.

[rlpb]

I disagree. I see shell syntax as an “inversion” of normal quoting rules. Normally quoting is the default but to achieve anything special you have to use an escape. Conversely, in shell, quoting is not necessary but any special character will likely have unwanted behaviour unless you quote.

This is handy for quick scripting as a natural extension of a command line interface. A counterexample might be an equivalent script using the Python subprocess module. For a simple script, the required extra quoting overhead is not worth the trouble. For something complicated it’s essential and a shell script is no longer suitable. For something in the middle, a carefully crafted shell script (eg. one that passes shellcheck) may be the most suitable solution depending on the situation.

Both cases have their place. It’s about using the right tool for the job.

[layer8]

Your `$foo` example confused me at first because I thought the backticks are part of the example.

[garethrowlands]

Sorry, the backticks were merely meant to delimit the code.

[assimpleaspossi]

I will immediately remove all the shell scripts on my system based on this comment alone.

Posted from my phone since my system came crashing down.

[garethrowlands]

Heh, I like that. We are indeed stuck with posix shell syntax; I didn't claim we weren't.

[assimpleaspossi]

I don't consider it being stuck at all. And that's coming from a die hard C and assembly language programmer.

[amelius]

Yes a large percentage of bash scripts fails if you feed them filenames with spaces or other special characters. Or a large amount of filenames near the Unix maximum commandline length.

Bash et al are great for command line use, but produce a situation of really bad engineering hygiene when used in scripts.

Do not use. Stay away. Bash et al considered harmful. Red flags on job interview when referenced.

[ndsipa_pomu]

Whilst I agree that Bash makes it far too easy to do it wrong, it's not too tricky to write scripts that don't have those issues.

Possibly the most important tip is to use ShellCheck (https://www.shellcheck.net/) as a linter on your scripts and work through the various warnings to eliminate them. For the rare times that ShellCheck is overly cautious, you can remove a specific warning with a comment before the offending line such as

  # shellcheck disable=SC1091

Also, take time to look through Greg's Wiki as it is possibly the best Bash resource that discusses the various pitfalls: https://mywiki.wooledge.org/BashPitfalls

As a Bash script writer, I disagree with your "red flag" interpretation as that'll mean that you'll end up with poorly written Bash scripts along with a hotch-potch of different tools to achieve the functionality (e.g. different versions of Python required for older scripts and newer ones). Knowing Bash is also very useful for writing Dockerfiles.