Hacker News new | ask | show | jobs
by teraku 2165 days ago
The concept of a cyclic workflow sounds interesting to me! I also intend mine to be acyclic, but I guess a possible infinite loop is not bad in itself. Maybe I should reconsider my underlying data structure.

Why would a workflow need to be static? I would say that a specific instance of a workflow should be static. The main reason one would want to switch to such a system is because they want their jobs to do (almost) the same thing, but at different points in time. If your task is also changing while it runs, maybe this logic should be within the task, not the workflow. But if your workflow changes over time, that is a very valid point. One which I'm also trying to incorporate with incremental versioning.

Backfills as a separate notion I also find weird.

Lastly I view the executor part as very important. Because imagine you want to run different processes across an organisation inside your scheduling engine. And some are written in a Python environment and others are compiled code. Sure you can schedule it all directly to k8s, but then you lose the advantage of bundling all your workflows into one system specified for that reason. You basically go back to your "cron" example, where you deploy directly on infrastructure. Meaning you never intended to use a workflow engine in the first place :P

Thanks for the input!