[hughesjj]

Data pipelines was also a perennial shit show behind the scenes. Turns out integration is a lot of complexity even if the service is "simple" in concept and doesn't need to be so real time.

I feel like data pipelines and swf have been replaced by step functions+event bus+lambdas/fargate. We've furthered abstractions over time, and that's a good thing.

Edit that said no idea how they scale in comparison