[teraku]

Can you specify with what you mean by Airflow introduced unnecessary and unhelpful constraints? I'm very interested.

I'm currently working on a standard format for defining such workflows [0] and my own scheduling engine, which hopes to be as non-imposing as possible. It's supposed to be a "cron" for task scheduling with a dependency graph. Only added thing is that you can specify on what kind of environment you want to run your tasks.

So I would appreciate if you could tell me what were things that annoyed you in particular. I use airflow at work and I can list a million, but I don't know exactly with what you meant with that sentence.

[0] https://github.com/OpenWorkflow/OpenWorkflow

[coverup]

Sure! To start, just fundamentally—why assume workflows are DAG-shaped? Why no cycles? Lots of real-world processes contain unscheduled repetition that arises at "runtime."

Or what if I can only find out what the rest of the workflow looks like once I'm halfway through it? Why must workflow definitions be static? No "decision" elements as in a flowchart?

Someone might read these complaints and think I'm asking for a programming environment rather than a workflow tool, and that's kinda my point :P

The "unnecessary" side is typically project-specific, but I tend not to need a separate notion of `backfill`, or any of the `Executor` functionality for distributed execution. I suppose if I needed to run stuff on multiple nodes I would just schedule jobs on Kubernetes directly.

[teraku]

The concept of a cyclic workflow sounds interesting to me! I also intend mine to be acyclic, but I guess a possible infinite loop is not bad in itself. Maybe I should reconsider my underlying data structure.

Why would a workflow need to be static? I would say that a specific instance of a workflow should be static. The main reason one would want to switch to such a system is because they want their jobs to do (almost) the same thing, but at different points in time. If your task is also changing while it runs, maybe this logic should be within the task, not the workflow. But if your workflow changes over time, that is a very valid point. One which I'm also trying to incorporate with incremental versioning.

Backfills as a separate notion I also find weird.

Lastly I view the executor part as very important. Because imagine you want to run different processes across an organisation inside your scheduling engine. And some are written in a Python environment and others are compiled code. Sure you can schedule it all directly to k8s, but then you lose the advantage of bundling all your workflows into one system specified for that reason. You basically go back to your "cron" example, where you deploy directly on infrastructure. Meaning you never intended to use a workflow engine in the first place :P

Thanks for the input!

[prpl]

cyclic workflows have strange rerun/retry behaviors. Lazy DAGs are implementable with subdags.