[andbberger]

slightly off topic, but there is dire need for a scientific "workflow manager" built to FAANG engineering standards attuned for the needs of academia (ie primarily designed to facilitate execution of DAGs on clusters). The airflows of the world have complex unnecessary features and require extensive kitbashing to plug into slurm and the academic side of things is a huge mess. Snakemake comes the closest but suffers from massive feature creep, a bizarre specification DSL (superset of python) and blurred resource requirement abstraction boundaries.

[slt2021]

Academia better to learn k8s and one of the k8s-native workflow orchestrators. This is as close to FAANG grade and open source as they can get, and arguably a bit better than this repo

[andbberger]

for better or worse slurm is the status quo for HPC. it works, every university has a slurm cluster, people already know how to use it

[torrance]

What about Nextflow?

[andbberger]

I considered Nextflow before begrudgingly settling on snakemake for my current project. Didn't record why... possibly because snakemake was already a known quantity and I was under time pressure or because I felt the task DAG would be difficult to specify in WDL. It's certainly the most mature of the bunch.

[_Wintermute]

Nobody wants to write or debug groovy, especially scientists who are used to python. It also causes havoc on a busy SLURM scheduler with its lack of array jobs (heard this is being fixed soon).