[b0afc375b5]

What about & and wait? Could it have been an adequate alternative?

[capableweb]

Probably for very simple use cases, but the real power in parallel really comes from the myriad of switches that enables so much more than what "&" and "wait" could do.

Here are a bunch of examples: https://www.gnu.org/software/parallel/parallel_examples.html

A fun one I end up using ~monthly or so for various things (usually with more switches added as needed):

    GNU Parallel as queue system/batch manager

    # start queue
    true >jobqueue; tail -n+0 -f jobqueue | parallel

    # add job
    echo my_command my_arg >> jobqueue

    # to start queue for remote execution
    true >jobqueue; tail -n+0 -f jobqueue | parallel -S ..

[klyrs]

When I'm using parallel, it's usually because I have thousands of jobs. Worse, they have nontrivial memory requirements. When you background processes with &, the system starts timeslicing. Each process gets to allocate its memory before being paused to make time for the next process. Your system will almost immediately crumple under load. Hopefully, the oom killer will target your backgrounded jobs... but the script spawning them will go untouched because it isn't the thing hogging memory.

Before I learned of parallel, I tried a hack where I'd manually assemble jobs into batches, and wait on the batches before starting the next. It achieved very low system utilization, because inevitably, one job each the batch takes much longer than the rest. A slight improvement (still not good), is to use `split` to chop your jobs file into $num_cores chunks, and background each chunk. But still, this gets low utilization. Problem being that you aren't using a thread/worker pool.

Parallel (or, TIL, xargs) can maintain 100% system utilization, until the very last $num_cores jobs.

[eisbaw]

No, that is more messy and can easily leave lingering processes.

But it can be done in pure BASH: https://gist.github.com/mped-oticon/b11dafa937e694ce4fa6fbf2...

GNU parallel supports expansion, which bash_parallel doesn't. However bash_parallel works with bash functions, which GNU parallel doesn't.

[untilted]

GNU parallel supports bash functions, provided you "export -f" them beforehand

[agumonkey]

You just taught me something