Hacker News new | ask | show | jobs
by zbruhnke 3250 days ago
Actually Lambda was my first thought as well.

I'll give you an example - When I first discovered Lambda I had been working with containers for 6-8 months already and when I saw lambda it hit me that they just have container images capable of running python and node projects that this spin this up in and run the code we ask them to.

My mind immediately then went to our data pipeline that I'd like to run on a nightly schedule. This is typically done using a tool like Airflow or Luigi but I thought I could probably even just build the whole thing in lambda tasks that trigger via cron and SQS.

The reason we ultimately did not do that with Lambda was because it limited the amount of A. time and B. Compute resources one lambda task could use and we did not want to try to optimize for that when working with increasingly large amounts of data.

With something like this (ACI) I could actually do exactly this and even if its triggered by something like Airflow or Luigi I could run those on a much smaller instance and save the larger compute problems to being done on ACI in small spurts

1 comments

FYI that while you can solve similar problems with Container Instances and Lambda, the closest offering to Lambda is Azure Functions.