[cyberax]

Spot prices don't directly reflect capacity. EC2 Spot used to be a real spot market, with actual real auctions where higher bids displaced lower bids.

But it was changed a while ago, so prices are now set algorithmically based on predicted demand/supply.

We run stateless calculations on EC2 across regions, and we definitely see that instances are harder to come by. Especially instances with GPUs. And for many instance types, the price advantage of EC2 Spot compared to committed spending is not significant anymore.

[MichaelZuo]

Huh, do you remember what the rationale for the change was?

[cyberax]

Well, I was one of the engineers that made the change :) I'm not sure how much I can tell, but the public reason was: "to make pricing more predictable".

Basically, one of the problems was customers who just set the spot price to 10x of the nominal price and leave the bids unattended. This was usually fine, when the price was 0.2x of the nominal price. But sometimes EC2 instance capacity crunches happened, and these high bids actually started competing with each other. As a result, customers could easily get 100 _times_ higher bill than they expected.

Another issue was that EC2 spot internally in AWS was implemented as a "bolted on" service that was not on the EC2 instance launch path, so it was easy to game the market. You could do very fun things, like:

1. You need an instance type that is right now under heavy contention. Not to worry! There's a way to get these instances!

2. You create a small VPC with only a couple of available IPs.

3. Then you submit a thousand EC2 Spot bids at 10x the price.

4. Your bids win and EC2 terminates other customers' instances. After all, you're willing to pay more!

5. EC2 then tries to launch these 1000 EC2 instances into the VPC. And fails, because there aren't any IPs available (see item 2).

6. Whoopsie. The bids are cancelled and EC2 instances are returned to the pool. Oh, and you're not charged anything because instances failed to launch.

7. Profit! Now there is plenty of capacity and you can submit bids at a normal price.

[MichaelZuo]

> Basically, one of the problems was customers who just set the spot price to 10x of the nominal price and leave the bids unattended. This was usually fine, when the price was 0.2x of the nominal price. But sometimes EC2 instance capacity crunches happened, and these high bids actually started competing with each other. As a result, customers could easily get 100 _times_ higher bill than they expected.

Thanks for sharing, though I'm confused about this part. It seems like expected behaviour?

Any possible spot/auction system I can think of would have the inherent possibility of a sudden surge in pricing.

[cyberax]

Sure. It was the correct behavior, inherent to any market system. Like Texas electrical spot market: https://theconversation.com/whats-behind-15-000-electricity-...

But it turns out that this is not a customer-friendly behavior. So AWS decided to remove the bidding out of the equation, and instead terminate instances based on a complicated scoring system.

The idea is that it's easier to deal with the missing compute capacity, which you notice right away, rather than be blindsided with a 100x bill at the end of the month.

[derefr]

I would guess they were seeing thundering herds of bots trying to fight to grab spot-instance stock whenever any appeared below the bots' set cost threshold. Basically like scalper bots.