Yeah, that was it! It was so confusing initially trying to figure out what should go in the EC2 instances' local disk storage, what should go into EBS, and what should go into S3. It all seems so obvious now of course...
It was also confusing to the users. I remember seeing a post every other week on HN about a service going down or losing data because they accidentally used the ephemeral devices for their databases. Haven't seen one of those in ages.
Yup, and for performance, you would RAID Stripe a bunch of EBS volumes together. But sometimes you'd get a volume on a noisy neighbor, and its performance would be terrible.
Where I worked at the time, I wrote some Chef tooling that would hdperf the volumes it provisioned, and if some weren't any good it would provision more until it got 8 good ones. Only then would it RAID them together, then deprovision the bad ones.
Now they have provisioned iops, and I haven't seen a slow EBS volume in a decade.