| >What is there to learn from an "storage industry expert" or major vendors? I mean, literally every problem they outlined. >Compounding this latency, hard drive performance is also variable depending on the other transactions in the queue. Smaller requests that are scattered randomly on the media take longer to find and access than several large requests that are all next to each other. This random performance led to wildly inconsistent behavior. Early on, we knew that we needed to spread customers across many disks to achieve reasonable performance. This had a benefit, it dropped the peak outlier latency for the hottest workloads, but unfortunately it spread the inconsistent behavior out so that it impacted many customers. Right - which we all knew about in the 90s, and NetApp more or less solved with WAFL. >We made a small change to our software that staged new writes onto that SSD, allowing us to return completion back to your application, and then flushed the writes to the slower hard disk asynchronously. So a write cache, which again every major vendor had from the beginning of time. NetApp used NVRam cards, EMC used dedicated UPSs to give their memory time to de-stage. Etc. etc. >network attached block level storage at AWS's scale hasn't been done before. This is just patently false. It's not like EBS is one giant repository of storage. The "scale" they push individual instances to isn't anything unique. The fact they're deploying more pods in totality than any individual enterprise isn't really relevant beyond the fact they're getting even greater volume discounts from their suppliers. At some point whether I'm managing 100 of the same thing or 1,000 - if I've built proper automation my only additional overhead is replacing failed hardware. Downvote away, watching HN think that re-inventing the wheel instead of asking someone who has been there already what the landmines are seems to be a common theme. |
Regarding a durable/asynchronously destaged write cache, I think EMC Symmetrix already had such a feature in the end of '80s or 1990 (can't find the source anymore).