[elsherbini]

Snakemake [0] is a tool worth checking out. You can use it to create declarative workflows, and similar to make, it creates a DAG of dependencies when you give it your desired output. Each rule can specify how many threads it needs and other arbitrary resources and the scheduler uses that to constrain execution. Workflows are architecture independent - you should be able to execute a snakemake workflow on a laptop, in the cloud, or on an HPC cluster.

It also allows you to use UNIX pipes with your dependent jobs when that is appropriate [1].

[0] https://snakemake.readthedocs.io/en/stable/index.html

[1] https://snakemake.readthedocs.io/en/stable/snakefiles/rules....