[peferron]

Just looking at the report's second graph, c5d.4xlarge has twice the throughput of i3.4xlarge, which is incredible. I wish AWS would release a new generation of "i" instances, with the same large amount of local storage as i3, but built on the same platform (including Nitro and processor choices) as c5/m5.

[malisper]

To be fair, an i3.4xl has 10x the disk and 4x the memory as a c5d.4xl at 1.5x the cost. If you are optimizing for GB/cost, i3s are still a reasonable choice.

There's also the i3.metal which has no hypervisor. I'm curious how the performance of one of those compares to a traditional i3.

[derefr]

> I wish AWS would release a new generation of "i" instances, with the same large amount of local storage as i3

I figure they can't. [Economically, I mean.]

I always figured the "local per-instance storage" on most instance-types was actually not literally local, but rather a set of disks allocated from a per-rack iSCSI disk server. (The host "wastage" if this wasn't true would be quite large.) The "i" instance-type hosts—especially the ones for the large instance-types—were likely just making a claim for the entire disk-server of their rack (leaving the rest of the rack to be schedulable only by instances that need no instance disks.)

Since iSCSI and "bare metal" don't go together, on Nitro, you'd actually need to build instances with real physical reserved disk pools that the hardware can see. That may even mean putting the disks inside the computer (shock horror!) and thus needing complex 2Us that need to be "recycled" (= having the still-good stuff fished out of them when they die) and may need to be opened up by ops folks for more than one reason, rather than just having "throwaway" compute blades + equally "throwaway" hotswap disk pools. In such a 1990s-reminiscent setup, i3.metal seems like a sensible upper bound for how big such an instance could get, economically.

Maybe you don't need the 2Us; maybe you can build the disk pool as something like a disk server (i.e. a dedicated PCIe backplane leading to RAID-controller daughterboards? something something Thunderbolt?) That'd lower the TCO of these host machines a bit, but it'd still be questionable how much usage such instances would get—and when they're unscheduled, despite the disks being a separate physical box, those disks would still be unavailable for any other instance to use, even ones scheduled to machines in the same rack.

(Or maybe they could get really fancy, and have a disk-server rack that can present itself as a RAID controller over PCIe, such that, most of the time, it can just be a disk server, but when it gets reserved by a Nitro-i-instance, it can switch off its Infiniband cards and switch on its PCIe client interface cards, and moonlight as a RAID array. If AWS does get bigger i-type instances, I'd wager that this is what they would have built to achieve that. That or custom RAID controller cards that present Infiniband-rDMA targets as if they were local NVMe devices, and don't allow host configuration, only BMC configuration.)

[cthalupa]

This would clash with the AWS documentation and re:invent discussions:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Instance... >This storage is located on disks that are physically attached to the host computer.

https://youtu.be/eWFEJmsddV0?t=1400

[peferron]

But c5d, m5d, r5d and other instance types use Nitro and have "local NVMe-based SSD storage" just like i3. Doesn't that contradict your entire point?

If you picked a r5d, and either gave it a larger piece of the local SSD pie or replaced the local SSDs with larger capacity ones, you would essentially get the i4 that I've been calling for. I'm sure it's more complicated than that, but I would be surprised if technical roadblocks were the main reason why this hasn't happened yet, rather than e.g. lack of customer demand.

[markonen]

AWS has already built that—on Nitro based systems, even EBS volumes are NVMe attached.

[altmind]

you can still have remote disks(same or next rack, 2m runway) with bare metal if you use HBA with external sas cabling.

this works reliably, native speed, not too expensive and is well tested.