[rektide]

> Today I would like to tell you about the AWS Nitro SSD.

A bit light on technical details but very fun, very exciting. Kind of sad that such amazing work is no longer quite so public, is no longer something that say Intel is going to talk up in endless details with a product launch. A huge amount of the work & innovation here is extremely specific, extremely private- all this Elastic Fabric Adapter related stuff is advanced systems engineering, close integration of systems, that's Amazon's & Amazon's alone.

Anyhow. This article pairs very well with the "Scaling Kafka at Honeycomb"[1], which I found to be a delightful read on adapting & evolving a big huge workload to ever-improving AWS hardware.

[1] https://www.honeycomb.io/blog/scaling-kafka-observability-pi... https://news.ycombinator.com/item?id=29396319 (38 minutes ago, 13 points)

[jeffbarr]

I wrote the AWS post and did my best to share lots of technical details; are there any specific things that you want to know more about?

[dmw_ng]

Generally a fan of your posts, but this one was very heavy on marketing buzzology ("cloud scale"). I can't tell if there was a genuine use case for designing a proprietary SSD, or if it were some pet project. Is "75% lower latency variability" because the first gen SSD was a CS101 project, or because AWS have developed some material edge over what others (with much wider scope) in the industry have been doing for years? I can't tell.

I can't see a reason to buy or use this product.

[jeffbee]

I doubt that other companies' supposedly "wider scope" actually exists or gives them advantages. Both Amazon and Google make their own SSDs and have the largest computer installations in the known universe. The fact that Samsung makes a lot of SSDs for laptops may not give them wider scope at all.

[simonebrunozzi]

I actually think that these posts have gotten much better over the past 2-3 years, at least based on my taste; the level of technical details is just right. On specific topics, I wouldn't mind James Hamilton-level specifics, but you can't be too deep on everything all the time.

(hi Jeff! Hope you're well :D)

[jeffbarr]

Hi Simone, doing well and we are trying to add more info while still being frugal with words and with the time of our readers.

[kaliszad]

Oh I would love some more deep dives or presentations by James Hamilton into various aspects of AWS. They combine the high level overview and the deep technical details in a very informative and entertaining way.

[rektide]

Hi Jeff! Eeeeeek! I'd love to know so much more about the Nitro acceleration. All these accelerated fabrics are so interesting.

* What does the Nitro accelerator look like to the host? . Does the Nitro accelerator present as NVMe devices to the OS host, or is there a more custom thing it presents as? Does the Nitro accelerator use SR-IOV to or something else to present as many different PCIe adapters, per-drive PCIe, or a single PCIe device, or no PCIe devices at all, something else entirely (and if so what)? Are there custom virt-io drivers powering the VMs? How much change has gone into these interfaces in the newest iterations, or have these interface channels remained stable?

* What is the over the wire communication? Related to the above; ultimately the VM's see NVMe, & how far down the stack/across the network does that go? Is what's on the wire NVMe based, or something else; is it custom? What trade-offs were there, what protocols inspired the teams? Originally at launch it seemed like there was a custom remote protocol[1]; has that stayed? What drove the protocol evolution/change over time? What's new & changed?

* What do the storage arrays look like; are they also PCs based? Or do the flash arrays connect via accelerators too? Are these FPGA-based or hard silicon? Are there standard flash controllers in use, or is this custom? How many channels of flash will one accelerator have connected to it? How much has the storage array architecture changed since Nitro was first introduced? Do latest gen nitro & older EBS storages have the same implementation or are newer EBS storages evolving more freely now?

* On a PC, an SSD is really an abstraction hiding dozens of flash channels. There have been efforts like Open Channel SSDs and now zoned namespaces to give the PCs more direct access to the individual channels. Does the Nitro accelerator connect to a single "endpoint" per EBS, or is the accelerator fanning out, connecting to multiple endpoints or multiple channels, doing some interleaving itself?

* What are some of the flash-translation optimizations & wins that the team/teams have found?

And simply: * How on earth can hosts have so much networking/nitro throughput available to them?! It feels like there's got to be multiple 400Gbit connections going to hosts today. And all connected via Nitro accelerators?

It's just incredibly exciting stuff, there's so much super interesting work going on, & I am so full of questions! I was a huge fan of the SeaMicro accelerators of yore, an early integrated network-attached device accelerator. Getting to work at such scale, build such high performance well integrated systems seems like it has so so many interesting fascinating subproblems to it.

[1] https://www.youtube.com/watch?v=e8DVmwj3OEs#t=11m58s

[Andys]

> * How on earth can hosts have so much networking/nitro throughput available to them?!

I feel this is something overlooked when people complain about the egress fees

[lend000]

If you have an existing EC2 instance with EBS storage and want to convert it to the new Nitro SSD, what will be the process for migration? E.g. a live swapping of attached storage devices, a quick reboot, or spinning up a new instance?

[jeffbarr]

The Nitro SSDs are currently used as instance storage, directly attached to particular EC2 instances.

[lend000]

Thanks for the response. To clarify, does this mean that only some EC2 instances will be eligible (i.e. if I have an older EC2 instance I will have to re-create it)?

[Androider]

Nitro SSDs appear to only be available on specific new instances types, like the just announced Im4gn and Is4gen.

[posnet]

Are there plans to provide Metal instances with these new SSDs?

[jeffbarr]

I don't know one way or the other, but great question. I prefer launching stuff to hinting about it :-)

[posnet]

Fair enough, and good luck with the rest of re:Invent

[sitkack]

I'd like to see P99.9 and MAX latency for certain read and write patterns. More concretely a before and after wrt a specific workload would be even better.

[ignoramous]

> A huge amount of the work & innovation here is extremely specific, extremely private- all this Elastic Fabric Adapter related stuff is advanced systems engineering, close integration of systems, that's Amazon's & Amazon's alone.

You speak my mind: https://news.ycombinator.com/item?id=19162376 (from 3yrs ago)

[b9a2cab5]

Intel has stopped disclosing a lot of details on their newer products, probably because they're no longer far and away the market leader. I think if AWS ever develops a 4-5 year lead over everyone else we'll see similar disclosures out of them. Facebook publishes a lot of info about Oculus asynchronous reprojection techniques and computer vision because they have a 2/3 marketshare in VR.