[jdcarr]

We don't know their memory usage but if they use 1.5GB then they'll get 70 hours/month free from Amazon when using lambda [1], and it gets better from there depending on their usage.

But seeing as most of their work last less than an hour, I wonder how well they'd done to take advantage of the spot market.

Does anyone know of any tools that take advantage of the spot market? I know that Spark does [2].

[1] https://aws.amazon.com/lambda/pricing/ [2] http://spark.apache.org/docs/latest/ec2-scripts.html

[manishsharan]

I have always been curious as to how to use the spot market. If I were to get a Ec2 on the spot market, how do I ensure that my task gets completed before being prempted? Mt batch jobs take anywhere between 10 ~ 20 minutes. I there a minimum guaranteed time that I could buy? For my solution , I ended up using Google cloud and its minute billing.

[vosper]

I lot of my infrastructure runs on the spot market. If you launch an individual machine there's no guarantee that at any time it won't be pre-empted (terminated). You do a little time to handle graceful shutdown of your code before the machine goes away, so it's not like the plug was pulled out at the wall.

If you're using Spot Fleets (a new feature) AWS will take a spec of the compute capacity you want and find it for you, across different instance types and availability zones, which makes it more likely you'll get the capacity you want.

In practice spot machines are pretty stable. We run hundreds across several different instance types and they're often up for weeks at a time.

So unless your batch job absolutely must not be delayed you might give spots a go. We used to run critical analytics EMR (Hadoop) jobs on spots and it saved us a a fortune - probably hundreds of thousands of dollars. Very occasionally we would switch them to on-demand (automatically) if we couldn't get spots at the price we wanted.

[vgt]

Have you looked at Google's Preemptible VMs? The big difference is that it's a flat 70% off fee, rather than a volatile market, and Google has relatively few but versatile VM types, which makes it easier to architect infrastructure (ex. no need for a high-IO VM or a high-network VM.. just attach a high-IO disk to a regular VM and networking is world-class).

[vosper]

We're heavily integrated into the rest of the AWS system, it wouldn't be worth the engineering effort. Also, the spots don't make up a large enough portion of our overall AWS spend, if we wanted to save money there are other things we could do that would let us stay on AWS (which we like).

[zbjornson]

You can do block request for one hour or more of guaranteed run time, but you pay for that guaranteed minimum run time. Otherwise you have to build in checkpointing and recovery.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requ...

[darkr]

AWS launched spot blocks last year that does just this. It's not as cheap as regular spot instances, but you do get a guarantee that your instance will be around for a configurable period between 1 and six hours (you pay more as you approach the six hour mark), after which time it will be automatically terminated.

I guess this is useful for longer-running batch jobs (longer than the 2 minute warning you get of termination on regular spots) of a relatively known duration.

https://aws.amazon.com/blogs/aws/new-ec2-spot-blocks-for-def...

[boulos]

Combined with per-minute billing and our lower on-demand pricing, regular GCE instances often end up being cheaper.

For example, a fixed duration m3.medium is currently $.037/hr for 1 hour and $.047/hr at 6 hours. GCE's equivalent n1-standard-1 is $.05/hr all the time (and automatically triggers sustained use discounts). If you ask for a 4-hour block and only use 3.2 hours you pay the same 16 cents as you would have on GCE. As I tell my friends, if you want a cheap, guaranteed VM just use GCE on-demand; if you're locked into AWS though, check out Spot!

Disclaimer: I work on Compute Engine (and Preemptible VMs specifically), so I'm clearly biased.