| This description, to me, sounds like you wrote a Python program... in shell. Nothing wrong with Python programs, but writing them in shell is about as pleasant as writing them in C. The shell’s parallel processing capabilities shine when you want consecutive stream transformation steps to run in parallel (note that you can pipe to and from loops, over multiple descriptors if necessary). When your task can be structured like that, it will often take as much time to write a parallel implementation in shell as it will a serial one in Python (etc.). On the other hand, running multiple iterations of the same step in parallel is, as you’ve seen, awkward. If the only thing you actually need to parallelize is HTTP requests, you might find piping things into aria2c[1] or the like to generate temporary files with responses and then processing those serially (best if your tool can tell you on stdout when a job is done, otherwise inotifywait and its ilk[2] may help if you’re uncomfortable just hammering the filesystem). When that is not enough, you could try GNU Parallel[3] or, if all else fails, implementing a jobserver in the style of GNU Make[4], which you may find more pleasant to do in shell than your current design. Generating a Makefile and then letting Make manage the worker pool (and call out to a separate shell script for each job) is also a possibility. But at that point (in particular if you’re not I/O-bound) it might be best to just not do it in shell[5]. [1] https://aria2.github.io/manual/en/html/aria2c.html [2] https://jvns.ca/blog/2020/06/28/entr/ [3] https://www.gnu.org/software/parallel/ [4] https://www.gnu.org/software/make/manual/html_node/POSIX-Job... [5] https://sanctum.geek.nz/etc/emperor-sh-and-the-traveller.txt |