Hacker News new | ask | show | jobs
by fwip 2361 days ago
You can definitely do parallel / concurrent / async programming directly in the shell, without any helper programs. It's not exactly ergonomic, but with a few patterns it's not too difficult.

Async:

    long_running_cmd & > results.txt
    cmd_pid=$!
    # other stuff
    wait $cmd_pid
    # do something with results, if you want
Parallel:

    for script in "${scripts[@]}" ; do
      "$script" &
    done
    wait
Dataflow:

    mkfifo my_channel
    mkfifo another_channel
    my_cmd > my_channel &

    tee my_channel \
      >(cmd1) \
      >(cmd2 > another_channel) &
 
    another_cmd < another_channel

But if you want a useful dataflow environment, I really recommend checking out [Nextflow](https://www.nextflow.io/). I use it at work for constructing bioinformatics pipelines, and it's really natural. The "preview" DSL2 is worth looking at, as well.
1 comments

Thanks you for the recommendation, Nextflow looks exactly like what I was looking for! Thanks for the bash examples as well.