|
|
|
|
|
by DocTomoe
1493 days ago
|
|
I think it's less about pure suitability, and more about availability. If I write a Bash script, I can be reasonably certain it will run on any unixoid system that came out in the last years. With Python, I am now in version hell (it's ridiculous how many Python2-first-servers I still find), I'll run the risk of includes suddenly becoming incompatible or buggy. Can Bash do everything that Python can? Almost certainly not. But it is available, it is relatively simple (and almost minimalistic), and it forces you to learn more about standard unix tools. Your specific use case ... well, I think that's pretty unusual - I'd wager if you had avoided the parallel processing and done it in a more linear way, you've had a better time. |
|
> if you had avoided the parallel processing and done it in a more linear way, you've had a better time.
I had to process ~500,000 files, and the aws api call was on average ~1 second, so it would have been a significantly longer time to process linearly. For example the bash version I whipped up processed ~30k files in 2.5 hours, while the python version did 30k files in ~20 minutes.
But yes I agree, if I didn't have to do such a large volume at once the bash version would have been just fine.