[tw04]

I think the most fascinating thing is watching them relearn every lesson the storage industry already knew about a decade earlier. Feels like most of this could have been solved by either hiring storage industry experts or just acquiring one of the major vendors.

[dekhn]

There are substantial differences in developing hyperscale storage from the systems that were built previously. But note that many of the architects of these systems were previously storage industry experts, and acquiring an existing vendor would not have been an asset to AWS since these new systems had a wide range of issues that the vendors never had to solve.

Your comment elsewhere about NetApp solving all known problems with WAFL. Hahahaha. Have you tried deleting a 5TB file in a filesystem at 95% capacity with snapshots enabled?

[tw04]

>Your comment elsewhere about NetApp solving all known problems with WAFL. Hahahaha.

I guess it's a good thing I didn't say NetApp solved "all known problems with WAFL" - but critical reading would've required you to respond to the content of the post, not provide an inaccurate summary to make a point that wasn't there.

What I DID say is that NetApp solved the issue of spewing random small writes all over the disk, resulting in horrendous performance on subsequent reads from spinning disk.sace reclamation takes a while. What's your point, assuming you had one? Because deleting a 5TB file on a filesystem 95% full with lots of snapshots is a daily workflow for... nobody? And if it is: there are countless ways to avoid that situation, but I assume you knew that too?

[jeffbee]

Or tried to create the 2147483649th inode on a filer and watched it try to repair itself fruitlessly for a month?

[szundi]

Probably the warranty and support was expired until you reached that inode count - those evil people

[jeeyoungk]

What is there to learn from an "storage industry expert" or major vendors? network attached block level storage at AWS's scale hasn't been done before.

[nyrikki]

Some of it, like random IOPS, spindle bias etc...was well known.

Well among implementers, vendors were mostly locked into the vertical scaling model.

I ran a SGI cluster running CXFS in 2000 as an example, and by the time EBS launched, I was spending most of my SAN architect time trying to get away from central storage.

There were absolutely new problems and amazing solutions by the EBS team, but there was information.

Queue theory was required for any meaningful SAN deployment as an example and RPM/3600 had always been a metric for HD performance under random.

Not that everyone used them, but I had to.

[tw04]

>What is there to learn from an "storage industry expert" or major vendors?

I mean, literally every problem they outlined.

>Compounding this latency, hard drive performance is also variable depending on the other transactions in the queue. Smaller requests that are scattered randomly on the media take longer to find and access than several large requests that are all next to each other. This random performance led to wildly inconsistent behavior. Early on, we knew that we needed to spread customers across many disks to achieve reasonable performance. This had a benefit, it dropped the peak outlier latency for the hottest workloads, but unfortunately it spread the inconsistent behavior out so that it impacted many customers.

Right - which we all knew about in the 90s, and NetApp more or less solved with WAFL.

>We made a small change to our software that staged new writes onto that SSD, allowing us to return completion back to your application, and then flushed the writes to the slower hard disk asynchronously.

So a write cache, which again every major vendor had from the beginning of time. NetApp used NVRam cards, EMC used dedicated UPSs to give their memory time to de-stage.

Etc. etc.

>network attached block level storage at AWS's scale hasn't been done before.

This is just patently false. It's not like EBS is one giant repository of storage. The "scale" they push individual instances to isn't anything unique. The fact they're deploying more pods in totality than any individual enterprise isn't really relevant beyond the fact they're getting even greater volume discounts from their suppliers. At some point whether I'm managing 100 of the same thing or 1,000 - if I've built proper automation my only additional overhead is replacing failed hardware.

Downvote away, watching HN think that re-inventing the wheel instead of asking someone who has been there already what the landmines are seems to be a common theme.

[tanelpoder]

I'm guessing that at cloud scale, more innovation & scalability is needed for the control plane (not to mention the network itself).

Regarding a durable/asynchronously destaged write cache, I think EMC Symmetrix already had such a feature in the end of '80s or 1990 (can't find the source anymore).

[abadpoli]

> whether I'm managing 100 of the same thing or 1,000 - if I've built proper automation my only additional overhead is replacing failed hardware

Hahahah surely this is a joke, right?

If it’s so easy and you already had solved all these problems, why didn’t someone already build it? Why didn’t you build EBS, since you apparently have all the answers?

[hodgesrm]

> Feels like most of this could have been solved by either hiring storage industry experts or just acquiring one of the major vendors.

VMware tried something like that with vCloud Air [0], which attempted to build a public cloud on the ESXi hypervisor with hardware and expertise sourced from storage vendors. It failed.

There were a number of reasons for the failure but perhaps the most prominent was that VMware vastly underestimated the amount of engineering necessary to build a multi-tenant public cloud. As the article pointed out you need to optimize along the entire I/O path from client request down to sectors on disk. It's a significant mindshift from running VMs over a SAN, which is where VMware came from.

(I was involved in the project in its later stages. These are personal observations.)

[0] https://en.wikipedia.org/wiki/VCloud_Air

[kccqzy]

For the right kind of people, it's tremendously satisfying for themselves to rethink and independently rediscover the best solutions. If you hire an industry expert that already knows everything, asking them to design the same things and write the same code again is not satisfying at all.

[hobs]

Ok, but we're running a business here not catering to the satisfaction of random programmers. The NIH part of this stuff is a bit weird.

[shermantanktop]

Turns out that if you don't cater to the satisfaction of programmers, you often kill the golden goose, and then you end up on the sidelines saying "oh, you shouldn't have done it that way" while someone else is succeeding.

People often talk about standing on the shoulders of giants. To extend that metaphor, the only ones standing there are the ones who climbed the giant. If you hire a college grad and stick them on that giant's shoulder, it's not the same.

[hobs]

Sure... but as an eng manager my job isn't to cater to your creative outputs in subordination to good products.

Craft comes before art.

[shermantanktop]

Craft? I’ll take craft any day.

The downward slide of most great tech companies starts with managers finally losing patience with the engineers. Check out Boeing as a recent example. It’s not about art or craft, it’s about treating people as more than widget producers.

[d110af5ccf]

Presumably your job is to get results. If catering to someone in some way is able to achieve that then it's a perfectly legitimate approach to consider.

[kccqzy]

Sure but that assumes the company itself has sufficient attraction that can overpower the lack of job satisfaction. Maybe you pay more. Maybe your brand is more attractive thanks to PR. Maybe you offer more perks. Maybe the product itself is more appealing. Maybe it's something else. You have to have some other advantage to retain people who aren't satisfied in their jobs.

[01HNNWZ0MV43FF]

I'd burn out quicker and have to charge more for my salary if the work was too boring

[exe34]

also, you can often discover new and better solutions if you don't know they're impossible.

[dilyevsky]

The answer to why your strategy most likely would have been a failure lies outside of technology domain