Back in my early days of data eng for ML pipelines, I stumbled onto Drake- and it opened my eyes to managing pipelines as DAGs. This pattern is supremely effective, and I try to teach anyone who might benefit.
Drake looks interesting, the visualization looks fun.
I don't see any evidence that it handles refunding of steps when transitive depended on code is changed, though. E.g., if script BBB.py includes CC, and CC is changed, then all steps transitively depending on BBB should be rerun. My make-booster specifically deals with that case.
I also expect drake to have a slow start, which slows development.
I don't see any evidence that it handles refunding of steps when transitive depended on code is changed, though. E.g., if script BBB.py includes CC, and CC is changed, then all steps transitively depending on BBB should be rerun. My make-booster specifically deals with that case.
I also expect drake to have a slow start, which slows development.