That would have to be the worst job in the world - keeping Lustre going as an Amazon service, with management that utterly lacks understanding and sympathy.
They’d have to pay me a lot of money to do it, that’s for sure. I’d love to see the disaster recovery plans. Every major Lustre site I’m aware of has had a data loss “incident” at some point in their history. It’s possible AWS has it all figured out with background backups and block device replication and whatnot, but I’m skeptical.
I have doubts based on my experience with Lustre and a certain understanding of AWS operations. I'm guessing they're going for the Iranian minefield clearing technique - get a mob of kids, hand them plastic keys to heaven (or RSUs), and march them through the field.
'Ephemeral' was/is the original Lustre design model. It was intended for high performance swap/scratch at Livermore with a short data lifespan - your higher priority bomb sim forces mine to roll out to disk and back in later, and that's it. Lustre, even today, isn't long term stable. The longer you leave data on it, the greater the probability of corruption - even silently.
Same thought here. I spent two-plus years debugging Lustre issues for a very small set of customers. It was an absolute beast. Build process was a compatibility-killing license-violating nightmare. Fell over at the slightest provocation, with little info to help figure out why. Provided no metrics to speak of, and the thicket of inter-related settings (especially timeouts) made effective tuning almost impossible. I'd guess that Amazon spent many engineer-years removing or rewriting significant pieces, and even more establishing the safe configuration envelope for what remained. Even then, it's probably a nightmare for the SREs (or whatever Amazon calls them) who have to keep it running.