[enriquto]

> As soon as you deal with concurrency, things either get hard to learn or get super messy.

Strong disagree. There is a language where concurrency is trivial and beginner-friendly. It is called POSIX shell with standard GNU tools. To run several independent commands in parallel, you print them and pipe to GNU parallel. If your concurrency has an arbitrary tree of dependencies, you print it in the form

    target : dependencies ; command

and then run GNU make.

Languages that make this simple stuff complicated, like Python, do so by choice, not by need.

[valenterry]

There is a reason why no one uses shell for anything more complicated.

E.g. start two long-running commands and then close the one of them depending on some later output of the other one or vice versa.

If that's not already complicated enough, we can easily add things like, execute a command A and then another other commands (B, C, D) and, only once all these are finished, run a cleanup command D. But run D always, no matter wether B, C, D fail (think of closing a database connection). And now treat this whole thing as yet another command E, that itself can be used in the same place as e.g. B or C or D before (= composition).

Shell is great for many common tasks, but I would never call it beginner-friendly, even less "trivial to use for concurrency".

[mhitza]

I think the reason no one uses shell for anything more complicated is related to the fact that:

1. It's not generally taught in CS classes, and people just search up enough syntax to be able to acomplish their task.

2. A lot of it's variable expansion/substitution are remiscient of Perl.

3. Small mistakes can have huge impact (everyone knows how often rm -rf of unintended directories takes place). Just for the fun of it, I suggest everybody reads at least once the Unix Haters Handbook https://web.mit.edu/~simsong/www/ugh.pdf

If I understood your task correctly, it can be done just nicely in the shell (after all, this is what I think the shell excels at). You'll see the unpleasant side of it when dealing with function arguments and slicing values and referecing specific ones. I don't guarantee it does what you want to the letter, but just my quick interpetration of your description.

    function demo() {
      first=$1
      last="${!#}"

      $first &
      wait

      for f in "${@:2:$#-2}"; do
        $f &
      done
      wait

      $last
    }

    function first() {
      sleep 5
    }

    function pingy() {
      ping -c 8 google.com
    }

    function last() {
      echo -----------------------------
      echo done
    }

    function inception() {
      demo first pingy pingy
    }

    demo first pingy pingy inception last

[enriquto]

> Shell is great for many common tasks, but I would never call it beginner-friendly, even less "trivial to use for concurrency".

Yet somehow beginners in my lab cannot get their hands off the shell and we have to politely guide them to other languages... But still, I insist that parallelism in the shell is very easy. How do you do that in Python, for example?

    # convert images from jpg to png, in parallel
    for i in *.jpg; do
            echo convert $i ${i/jpg/png}
    done | parallel

Regarding your "complicated" use cases, I must confess that I don't understand them nor their relevance. But for simple people like me without fancy needs, the simple concurrency offered by GNU parallel and GNU make is good enough.

[valenterry]

Haha, please don't get me wrong! I didn't mean to say that concurrency in python is any easier - it's probably even harder than in shell.

> Regarding your "complicated" use cases, I must confess that I don't understand them nor their relevance.

They are two recent examples of mine. But even if you don't see the relevance in them: if shell truly makes concurrency trivial, then the shell code should not look much more difficult then my English descriptions.

[refactor_master]

Parallel independent batch jobs are easy in Python though. If you can write it as map(function, iterable) you just need to import the right module and drop it in and you’re done.