Any plans to implement Fargate as an option? You mention the limitations of Lambda and Fargate pretty much takes care of all of those, without needing to provision EC2.
Fargate is more of a maybe for us as it doesn't seem to offer a ton of advantages over EC2. It still takes about 30 seconds to launch a Fargate job, and as far as I can tell there's no way to "keep an instance around". With Meadowrun-on-EC2 or Lambda, when you run two jobs run one after another with the same libraries and the same code (or even slightly different code), there's almost 0 overhead for running the second job. So Fargate is only slightly better for a cold start (30s compared to 45-60s for an EC2 instance in my experience), and significantly worse for a warm start (still 30s). And that's the core experience we're trying to make amazing--run some code, look at the results/data, tweak it, run it again, repeat.
Meadowrun is taking care of all the messy details of provisioning and managing the EC2 instances, so Meadowrun-on-Fargate won't be any easier to use than Meadowrun-on-EC2, and I don't see a ton of advantages to make up for the inability to get a warm start on Fargate. That said, AWS is super dynamic, so we're definitely keeping an eye on Fargate.
Fargate is more of a maybe for us as it doesn't seem to offer a ton of advantages over EC2. It still takes about 30 seconds to launch a Fargate job, and as far as I can tell there's no way to "keep an instance around". With Meadowrun-on-EC2 or Lambda, when you run two jobs run one after another with the same libraries and the same code (or even slightly different code), there's almost 0 overhead for running the second job. So Fargate is only slightly better for a cold start (30s compared to 45-60s for an EC2 instance in my experience), and significantly worse for a warm start (still 30s). And that's the core experience we're trying to make amazing--run some code, look at the results/data, tweak it, run it again, repeat.
Meadowrun is taking care of all the messy details of provisioning and managing the EC2 instances, so Meadowrun-on-Fargate won't be any easier to use than Meadowrun-on-EC2, and I don't see a ton of advantages to make up for the inability to get a warm start on Fargate. That said, AWS is super dynamic, so we're definitely keeping an eye on Fargate.