[jason_slack]

I use GNU Parallel for pulling stock data from various sources, massaging it, creating flatfiles of the data, creating models of the data, etc.

I also use it as a rudimentary queue system for stacking up the next jobs (while scripts stack up the next jobs, but..).

It had a bit of a learning curve because the docs are really technical and not geared towards new users enough, but reading and re-reading and trying some examples helped cement.

Here are a few ways I use it:

echo "Number of RAR archives: "$(ls .rar | wc -l)

ls .rar | parallel -j0 1_1_rarFilesExtraction

ls -d stocks_all/Intraday/*.txt | parallel -j${ccj}% 1_2_stockFileProcessing {}

I'd like to scale this to work with multiple machines (as Parallel can do) but I get really tempted to just write my own parallel processor just to rely on my own code.