|
|
|
|
|
|
by mtrn
2868 days ago
|
|
|
Luigi is a great Python library for building multi-stage pipelines with parameterized tasks. It easily extend to new storage types and targets. We currently use it to build a moderately complex product from dozens of data streams (files, APIs and things in-between) and millions of records. At its core is a DAG and topological sort, which capture the essence of pipelines and execution. A DAG is easy to visualize, even its development over time[1]. [1] https://imgur.com/a/RuC0B5Y (generated from the projects' code, in this case https://github.com/miku/siskin) |
|
|