For example, in ETL pipelines, I would greatly prefer to have an entire DAG go down quickly and noisily than to risk having it generate incorrect data. It's the difference between a crappy morning, and a crappy day or even a crappy week.
Just curious since a lot of my work is on an ETL - what are your favorite DAG libraries/approaches? Some of my ETL workflows can run in parallel with each other because there aren't data dependencies between them, and others should definitely crash loudly
We use a home-grown solution, and try to keep it pretty minimal. For example, we can run all our pipeline steps serially and still get all the work done plenty fast, so, for now, we're keeping that parallelism can of worms firmly shut.
It's also viable because it's the convention. Library calls are likely to return {error, {the bad data you sent}} so it's easy to write receive {error, _) -> {missiles, {launch false}} end.
And, the fact that an Erlang server is explicitly intended for handling zillions of tiny, independent requests. A request fails? Meh. They'll call again if it's important. What matters now is the next tiny, independent request.
For example, in ETL pipelines, I would greatly prefer to have an entire DAG go down quickly and noisily than to risk having it generate incorrect data. It's the difference between a crappy morning, and a crappy day or even a crappy week.