[indoorskier]

Hi HN,

We're two programmers who have worked in core/platform engineering roles for most of our working lives. During that time, one of the main problems we've solved time and time again is to let people run their ad-hoc jobs and scripts on remote compute without hassle.

To solve this once and for everyone, we made Meadowrun, an open source tool that automates the tedious details of running Python code on cloud VMs. It runs in your AWS or Azure account, nothing else required.

No need to mess around with containers, SSH into remote machines, copy code across, set up images or look up instance types that sound like Starbucks orders ("t3.venti.oatmilk.latte") and what they cost.

All with the same experience as you'd have running on your laptop - just change the code or dependencies locally and run - meadowrun takes care of the rest.

We welcome any and all feedback!

[by_the_bay]

If your target workflows are ad hoc jobs, wouldn’t running on lambda make more sense than dealing with the headache of provisioning ec2?

[hrichardlee1]

Yes! (in some cases) The tradeoffs are that lambdas are way faster to start up, but they max out at 10GB of memory, 15 minutes of runtime, you can't use GPUs, and you get 250MB to store your code (you get more if you build a container) and 512MB of temp space. We're in the middle of working on adding this as an option--we want to make it seamless to switch between running your code on Lambda (for short, quick jobs) vs EC2 (for longer, more resource-intensive jobs) depending on what kind of workload you have.

[r00tanon]

Good news. We've found lambdas orchestrated with step functions and properly split out as to function can do a lot of data processing even with the limits you stated before needing to switch over to AWS batch or ECS. Nice to have that tool in the belt. I'll check it out.

[by_the_bay]

Cool, look forward to trying it out

[yunohn]

Their landing page claims that Lambda support is planned.

But I think there’s scope for both, data jobs needs ec2.

[binaryblitz]

This is an awesome idea!

Any plans to implement Fargate as an option? You mention the limitations of Lambda and Fargate pretty much takes care of all of those, without needing to provision EC2.

[hrichardlee]

Thank you!

Fargate is more of a maybe for us as it doesn't seem to offer a ton of advantages over EC2. It still takes about 30 seconds to launch a Fargate job, and as far as I can tell there's no way to "keep an instance around". With Meadowrun-on-EC2 or Lambda, when you run two jobs run one after another with the same libraries and the same code (or even slightly different code), there's almost 0 overhead for running the second job. So Fargate is only slightly better for a cold start (30s compared to 45-60s for an EC2 instance in my experience), and significantly worse for a warm start (still 30s). And that's the core experience we're trying to make amazing--run some code, look at the results/data, tweak it, run it again, repeat.

Meadowrun is taking care of all the messy details of provisioning and managing the EC2 instances, so Meadowrun-on-Fargate won't be any easier to use than Meadowrun-on-EC2, and I don't see a ton of advantages to make up for the inability to get a warm start on Fargate. That said, AWS is super dynamic, so we're definitely keeping an eye on Fargate.