[bhaak]

How about you don't? Bash as scripting language is rather mediocre.

Anything that is not simple in bash gets hard to read and debug and probably is wrong on some subtle levels.

I have a rule of thumb that any shell script that grows beyond a screenful of lines gets redone in a proper scripting language.

[periegetes]

Bash as a scripting language is actually pretty amazing. It gives you everything you need to perform some quick-and-dirty tasks with minimal overhead. If you need only work on sequential data, files and processes, it's a perfect match.

It's not a full-fledged programming language by any stretch of the imagination (lacking structures more complex than associative arrays), but it's damn good for scripts of all sorts.

As an example, I've reimplemented a subset of Ansible (a command able to send "modules" on multiple machines via SSH and capturing+caching their output for subsequent queries) in ~150 lines of Bash. Considering that the size of Ansible, written in the more proper Python, is ~15000 LOC, I'd say Python is the much lesser scripting language.

Edit: to answer the OP's question, the documentation I've found most helpful to learn Bash is the one present on the Linux Documentation Project, with the page for arrays deserving special mention : http://tldp.org/LDP/abs/html/arrays.html. I spent a lot of time reading the manual before stumbling upon that documentation, and none of it really clicked until I had a few examples before my eyes.

[ribs]

Typically a program like Ansible will have the majority of its use cases implemented in a minority of its code, while the rest of the code will be there to support special cases, edge cases, rare use cases, etc. So that contributes to the disparity in size.

[dkns]

Ansible has to take into account numerous edge cases, operating systems, backwards compatibility, etc. etc. Of course it's much, much bigger than your 150 lines bash script.

[barrkel]

OTOH, if you can structure your problem as a composition of pipelines, it can be quite a bit faster in bash than in a "proper" language. You get to choose optimized tools, and they run concurrently.

Writing efficient bash code forces you to think about your problem differently. It's a very similar process to thinking functionally; e.g. you don't want to deal with lines of a file one at a time in a loop, you want to do filters and maps in languages like grep, sed and awk to deal with data in a streaming fashion with a minimum of forked processes.

[fiatjaf]

Is there a "proper scripting language" somewhere that supports the same first-class access to Unix programs and the same piping | syntax that shells in general do?

I would love something like that.

[kronos29296]

Perl was created to do this stuff. Perl is like duct tape to hold together everything. Most people call it ugly because of that but Perl is the best language for quick dirty one liners that work. Only problem is that if you look at it later you will have problems understanding it.

[TheGrassyKnoll]

Agree. Perl is terrific for quick text processing. You can make surprisingly powerful one-liners.

[soveran]

In most platforms you can install the rc shell from plan9. It's what I use exclusively for my shell scripts. It is only when I want to share some script with other people that I consider using /bin/sh, and even then I've gone for rc nevertheless. Here you can read about it: http://doc.cat-v.org/plan_9/4th_edition/papers/rc

[chriswarbo]

Thanks for the suggestion. I've also been after a saner shell, and have been disappointed in one way or another with the approach of either using another language idiomatically (Python, Scheme, Haskell, Scala, etc.), since running commands, piping, etc. are all rather awkward; or using such languages with embedded shell-like libraries, which seem to have awkward edge-cases.

As a "real" shell, it looks like rc maintains the command, file and piping niceties of bash, whilst avoiding the edge-cases of shell-like embeddings.

[jerf]

In Perl, IPC::Run [1], while a bit on the quirky side sometimes, can do that sort of thing.

    use IPC::Run qw(run);
    IPC::Run::run(["echo", "abc"], "|", 
                  ["tr", "a-z", "A-Z"], "|", 
                  ["cat"],
                  \$out);
    print "out: $out";

will yield

    out: ABC

Useless use of cat here just to make a point, of course.

It's easy to use Perl strings as input or output, or files, and there's also ways to interact with streams as they go along.

Generally speaking the slightly more verbose interaction with shell is made up for by the savings when you can use Perl directly to do something rather than spawn a shell process for something simple. (Shell composition can be powerful, yes, but on the other hand spawning a full process and setting up a pipe for things like "tr" or "wc" is often just silly.)

I also personally believe there's a win in the syntax; you may say "What? 'tr a-z A-Z' is way more convenient than '["tr", "a-z", "A-Z"]' but I say one of the biggest and most pervasive errors in shell is to incorrectly set up the arguments by having something interpolated in incorrectly. Having an unambiguous syntax for separating arguments has often made it much safer for me to write the code that has to use the shell. Used correctly it can even be made to work in the face of user-supplied input, something that should generally not be combined with any form of implicit argument separation. (Although bear in mind that "used correctly" encompasses more than just "separating the arguments correctly.")

I am also NOT claiming exclusivity; this is just the thing I know off the top of my head. I'm sure the other scripting languages have nice libraries, too. Though watch out for them being too nice. There's a bit of an impedance mismatch between shell and scripting language. Anything that smooths it over too well is probably either giving up critical features or introducing subtle bugs, which can become security bugs in the face of arbitrary user input.

[1]: http://search.cpan.org/~toddr/IPC-Run-0.96/lib/IPC/Run.pm

[reacweb]

Perl of course, even if the adjective "proper" seems weird for Perl ;-)

As a rule of thumb, if a script starts to need arrays or to handle spaces in filename, I migrate it to Perl.

Error handling is also easier and more natural in Perl.

[fiatjaf]

Just found out that xon.sh http://xon.sh/tutorial . html does this.

[yellowapple]

Tcl lacks the piping syntax by default, but there are various ways to implement it: http://wiki.tcl.tk/17419

[oblio]

Then you have the age old problem: what if one of the tasks in your pipeline(s) fails?

[yorwba]

  set -o pipefail -o errexit

Add more options to taste.

[Benjamin_Dobell]

https://stackoverflow.com/questions/29532904/bash-subshell-e...

Not pipefail as such, but Bash's error handling semantics (or lack there of) is pretty lame.

[pjc50]

Indeed. While you can get the rules regarding file name escaping right, the language does nothing to help you get it right every time, so unless you're careful someone will leave a file with a space (or, God help you, a newline) in the name in exactly the wrong place and blow it up.

[amelius]

Totally agree.

Also, even if you manage to become better in Bash, you are bound to lose your skills at some point when you have been programming in other languages for a while.

I always have to look up how to do even basic things in Bash. I just don't use it often enough for these things to "stick".

[milkytron]

It's okay to not have everything memorized. As long as you know what to look for, you're only a google search away.

[jalanb]

How about OP does anyway? Bash as a scripting language is fucking great!

Since when is simplicity an argument against writing programs? Whether scripts or frameworks? "Hard to read" is not neccessarily an inherent trait[1] of the language, and more likely wrong on some PEBKAC level.

I have a customised environment at near 10k lines of bash in 5 projects, all of it in the correct tool for the job, aka a proper scripting language, so I can suggest another use for your thumb :-)

1: https://www.reddit.com/r/commandline/comments/2kq8oa/the_mos...

[gpderetta]

I would say that a screenful is already on the large side. Then again, there is a lot you can do in a screenful in bash.

[radiospiel]

If you have really limited resources, then bash or one of the other shells are the only tools at hand. Embedded devices might not give you enough disk space to get perl, ruby or python.

[Diti]

There is this one case for which shell scripting is particularly interesting: dependencies simplification.

https://github.com/EtiennePerot/parcimonie.sh/blob/48044f913...