[schemescape]

Slight tangent: is anyone aware of a library (in Python or JS… or C) for specifying DAG pipielines that supports caching results (based on input)?

Make is closest I’ve found, but there’s probably something better out there.

Edit: I should look at other build tools, but ideally I want a library.

Edit again: I will be running this locally. Forgot to mention that.

[tjfl]

Maybe Dagster with Assets?

https://docs.dagster.io/concepts/assets/software-defined-ass...

[brahbrah]

fn_graph: https://github.com/BusinessOptics/fn_graph

(My favorite one ^)

pyungo: https://github.com/cedricleroy/pyungo

Loman: https://github.com/janushendersonassetallocation/loman

Tributary: https://github.com/timkpaine/tributary

[mstokholm]

alkymi (https://github.com/MathiasStokholm/alkymi) is such a library (disclaimer: I'm the author). I built alkymi to help me cache complex data pipelines, while still having all the niceties of Python.

[schemescape]

I haven’t dug too deep yet, but this sounds exactly like what I was looking for, thanks!