[jwhite]

I spent many years avoiding shell, in large part because all the arbitrary-looking syntax and features (like quoting rules) the speaker highlights make it frustrating and difficult to master, and I felt like it was a treacherous enemy to be overcome through desperate struggle, rather than a tool and ally to rely on. I know some people thrive on mastering complex and intricate rules, but that's not my forte. I do better with overarching principles and a unified set of orthogonal constructs, such as C (ok, debatable, but compare to C++), Lisp, Python or Haskell.

It took a lot of pushing from a mentor for me to finally click with the overarching principles of shell and see how to master it and become productive with it. He really forced me into it as well, I wasn't a willing pupil for quite some time, but he kept persisting and finally I started listening to him. I'm so glad I did. I've never read any book or tutorial or guide that teaches what he taught me either, despite searching several times. Also, I don't think I could have learned to master shell if I hadn't learned functional programming first. The day I realised how amenable shell is to higher-order thinking was a real epiphany.

These days shell is a really important tool in my toolkit, and I can see just how much some of my colleagues are held back by not knowing it.

Anyway, I really enjoyed the video, it was a nice exposition.

[chubot]

I agree on both counts. My first experience with shell was the fact that 'foo=bar' works for assignment but 'foo = bar' doesn't. Surely that must be a mistake! Why would anyone design such a silly language?

I got past all of that, and I also see people struggling to accomplish things that can be done in 30 seconds with shell. My main language is Python, and for certain tasks it just can't compete with shell. Believe it or not I think I once wrote a Python script to do what "find | sed -i" does.

Now I'm back in the "who would design such a silly language" mindset, and I am designing and implementing a new shell. You might like some of my blog posts:

"Pipelines Support Vectorized, Point-Free, and Imperative Style"

http://www.oilshell.org/blog/2017/01/15.html

"Shell Has a Forth-like Quality" http://www.oilshell.org/blog/2017/01/13.html (on HN previously)

[jwhite]

Thanks for those links, I think I will enjoy them. I was actually thinking about Forth as well when I was writing those comments about functional programming. I'll be interested to read your take on it.

[coldtea]

>These days shell is a really important tool in my toolkit, and I can see just how much some of my colleagues are held back by not knowing it.

Perhaps, but the shell is still a badly accrued (more than designed) set of options.

[falcolas]

Well, on the plus side of that, accrued means they're actually useful, as opposed to "designer through they would be useful".

I wouldn't mind a better shell, but I'm not going to throw out my current shell for the lack of a better shell.

[jwhite]

Yes, totally agree. But it offers something different to say Python, as the video points out. Some things are more compact in shell than in Python, because it's not the same sort of programming language as Python. And I reach for Python as well, when it's the right tool.

[coldtea]

True. I'm not against a DSL for working with CLI programs, pipes, output streams, processes, etc. Just that the shell as we know it is not the optimal (or even close to optimal) example of that.

Something like ZSH and the modern Fish shell showed how much traditional shells can improve, but we also need some better primitives, and more controlled experience.

Oh, and they should NOT be based on emulating 40+ year old teletypes anymore, except as part of a "legacy" mode.

[catdog]

Totally agree, the concept is nice and really useful but its implementation is for a large part really a big pile of fragile and confusing hacks. Most of the power comes of powerful tools like find, sed, awk (being a programming language on its own), etc. The language gluing them together is nicely compact but apart from that very disappointing. Also the everything is a string approach has its limitations. A python script may sometimes be longer but a lot less riddled with annoying corner cases.

[ams6110]

I'm the opposite, I avoid GUIs. I use pretty much only the shell, and text-oriented programs. I use a graphical desktop just to manage my terminal windows. The only other application I use daily is a web browser.

I guess maybe because I started with computers before there were GUIs that is what felt normal.

[thomastjeffery]

> I guess maybe because I started with computers before there were GUIs that is what felt normal.

For me, probably not. I did use DOS before Windows as a kid, but that was long before really learning to use a shell. Familiarity with DOS simply made me familiar enough with the idea to be unafraid of it.

When I learned to manage files, handle software, etc. I learned to do it with a GUI in Windows. When I started to use Linux, I was vaguely aware of the usefulness of a shell, and took the opportunity to really learn. To this day, I rarely even have a file manager installed, in favor of a shell environment.

[jwhite]

Ha! I should have been more precise in my language. I was avoiding learning how to write more substantial shell scripts and programs, not the command line per se.

I also got my start on pre-GUI computers (a VIC-20, specifically).

[yjftsjthsd-h]

I'm confused. You had a good time with C but trouble with shell? I was strongly under the impression that 90% of Unix shells were strongly inspired by C semantics. Return codes, quoting, strings in general, etc. You maybe could even try csh if that was better.

[jwhite]

Every language has its warts, but C is a pretty small language when it comes down to it. Compare that to just the number of different ways you can expand a parameter in bash:

https://www.gnu.org/software/bash/manual/html_node/Shell-Par...

That's the sort of thing that turned me off. I still tend to avoid those nooks and crannies, although they are sometimes useful. I'm not an expert.

I'm not sure about Unix shells being inspired by C semantics. I actually started out on csh or tcsh, I forget which, two decades ago. I'm not sure that making a shell language "C-like" is a useful goal, and those two shells both fell out of favour because they have many flaws. I switched to bash quite quickly.

[nsomaru]

Care to document some of those insights for those of us in a similar position?

[jwhite]

Here are a couple of thoughts:

- I used to spend a lot of time stuffing things into variables and then trying to operate on those, which is how you do things in imperative languages. Shell is better thought of in dataflow terms, i.e. pipelines. So I started using pipelines more and more.

- The corollary is that you start thinking more about data (passing between pipeline stages) and less about control flow.

- Compare an if statement in C, for example, to an if statement in a functional language. In C an if statement has no value per se, but in functional languages if "statements" are actually expressions that have a value:

f :: A -> String f x = if isFoo x then "foo" else "other"

In shell an if statement is just another command, so you can do things with its "value", i.e. its stdout:

    if do_test
    then
        cat file1.txt
    else
        cat file2.txt
    fi | grep pattern

Because I was so used to thinking of if statements as just control flow, I never used to think of piping them to another command like that.

Also if you think carefully, that last pipe to "grep pattern" has the flavour of partial evaluation in functional programming.

- I tend to think more in terms of building up an execution environment these days, in the same way that lexical scope works in regular programming languages. In shell, of course, that means command invocations, sub-shells, etc., because the shell doesn't really have good lexical scoping. The simplest example I can think of is executing some commands in another working directory with a sub-shell:

    (
        cd $dir
        run_test > "${f}.log"
    )

You could compare it to RAII in C++ or with- style macros like WITH-OPEN-FILE in Common Lisp.

Those are just some examples.

EDIT: Also, have a read of the links in chubot's post: https://news.ycombinator.com/item?id=14012453 . That's the sort of mentality that will change your approach to shell programming.