Hacker News new | ask | show | jobs
by barrkel 4700 days ago
Need to download a bunch of files from a remote host, process them, and write them to disk?

    cat urls.txt | while read -r url; do 
      base="$(basename "$url")"
      wget "$url" -O - -o "$base.log" | process > "$base" &
    done
    wait
For ad-hoc scripting, I tend to prefer bash, with anything remotely corresponding to heavy lifting assigned to the language with the best library / performance / whatever critical factor for it. Fork-join parallelism in bash is very easy.