[simonw]

First impression: this is a brilliant piece of software design.

The ability to compose a map/filter chain and execute it in parallel against every object in an S3 bucket that matches a specific prefix - wow.

The set of problems that can be quickly and cheaply solved with this thing is enormous. My biggest problem with lambda functions is that they are a bit of a pain to actually write - for transforming data in S3 this looks like my ideal abstraction.

[koolba]

... Except it's not!

The "lambda" here isn't AWS Lambda. It's a locally executed function.

Now if this scheduled a bunch of real Lambdas to execute the work for each bucket then yes that'd be awesome.

[kot-behemoth]

It should be fairly easily doable with Gordon (https://github.com/jorgebastida/gordon), and scheduling via CloudWatch Events. Or Airflow.

[simonw]

Bah. My first impression was totally wrong in that case. Here's hoping someone builds a version of this that executes magically in the lambda cloud.

[illumin8]

Well, you could run it on a large EC2 instance (x1.32xlarge?!:O) and it would be running the lambdas on the cloud, technically... ;-)

[glogla]

That would be glorious.

[rpedela]

Yes, I concur! I am definitely trying this out. I have a couple use cases where I think lambda functions would be useful but I don't currently have the time to figure out how to write and execute them.

[wellsjohnston]

Thank you :)

Writing this was a necessity for me, being a 1-person data team coming from a Node.js background.