[cookiecaper]

I mean, it's a nice idea in theory, but in practice stuff finds its way onto the ephemeral disk even if you have EBS volumes mounted, and "Sorry, we just deleted all your crap, I guess you should've had that on EBS" (which is an extra fee by the way) is not an acceptable solution to the problem.

Yes, it would mean holding the hardware in reserve for the cooldown period. I'm not talking months here, just enough time to recover from an accidental "sudo shutdown -h now" instead of "sudo shutdown -r now" (or similar). It'd be nice if Amazon sent an email warning about the condition and giving you an hour or so to go in and save your data/restart your instance. They could even make it a policy that you're charged for the time your instance is running + 1 hour to facilitate cooldown feature if they're really that worried about it; it's better than wiping data as soon as someone stops (from AWS console) or shuts down (from real console) an instance and providing absolutely no avenue for recovery, no matter how quickly you notice the mistake.

[NathanKP]

I have never had stuff accidentally find its way onto ephemeral storage. The ephemeral storage is mounted at a specific location of /mnt. Everything else on the system (OS, binaries, application code and resources) is stored on an EBS volume.

You have to specifically put something into the /mnt folder if you want it to be stored on the ephemeral storage. Any other location is safe and will persist through halts and stops.

In practice the only thing you should ever use the /mnt folder is maybe a Nginx disk cache, or as an alternative /tmp or something like that. Basically if stuff you don't want to lose is finding its way onto the ephemeral storage then you are doing something wrong.

[cookiecaper]

It depends on your users. We have some users that are not super well-versed in AWS and they just see a big disk and put data there, and someone has to come back and move it to an EBS volume to make sure it's safe. /mnt is also used as a staging area for large files and the intention is always to move them to permanent storage when done, but that sometimes doesn't happen. /mnt is usually, in the non-AWS world, where the bigger, more authoritative disks, like an NFS mount to the NAS, would be mounted, so it's counter-intuitive to tell users to treat /mnt like /tmp. Even if someone is using /mnt as a temporary store because they understand EBS v. ephemeral, if they shut down from within the instance, they don't see any warning about the doom of the ephemeral data, and it may be unclear that a shutdown/system halt is the same as a "stop" in the AWS console, and they could lose the data that they had in the staging area unexpectedly.

There are plenty of plausible situations where an AWS user can find themselves with important, even just temporarily important, data on ephemeral. Whether those are the result of "correct" usage or not, it's beyond the pale to just zap that data away and tell the customer tough titties as soon as a shutdown command is issued.