[ian_d]

Nice tool, thank you for releasing it.

Unfortunately, in my experience it's a pretty normal strategy to over provision EBS gp2 volumes to get the IOPS since they scale linearly up to ~10k. I think that's the break-even point where it's actually cheaper to switch over to provisioned IOPS volumes.

So we'd have a number of little-utilized (in terms of data actually stored) 3TB drives just to avoid paying for provisioned IOPS.

[prakashmanden]

It is certainly a good strategy to over provision to achieve performance and reduce cost. We recommend it. Wrote a blog on it a while back which is here - https://www.fittedcloud.com/blog/is-it-possible-to-use-ebs-g...

[ckdarby]

I recommend getting off EBS and switching over to the i3 instance types. AWS pricing for the IOPS is crazy on EBS.

[a012]

the downside of i3 is you'll lose your data when instance is stopped, then you have to have another place to keep this data permanently.

[inopinatus]

Fun story: I once heard of a startup that was keeping all their data on ephemeral local storage, mirroring it between instances across AZs for "high availability", and then, er, using spot pricing for all instances to save money.

One day the spot price skyrocketed and all their databases got shot down.

And that, I presume, was the end of the company.

[a012]

for a second, I've thought it's me, except I don't use spot instance. I use i3 for IOPS with EBS backing it up

[RantyDave]

Have you considered getting a single large volume and sharing it over NFS? If this is a bad idea, why?

[prakashmanden]

Another option is to use EFS (AWS version of NFS). It's easy to use/set up. However, EFS costs 3x-12x (gp2-sc1) more than EBS. https://www.fittedcloud.com/blog/aws-elastic-file-system-a-q....

[jimktrains2]

Nfs can be slow and have consistency and locking issues under heavy write loads. Also security isn't often top notch.

It really depends on the use case.

[jimktrains2]

Instead of downvotes, please explain why I'm wrong. Not only do I want to know, but it's helpful to other.

These have been my experiences with nfs. It's great for certain things, especially read heavy ones. It's very not great at write heavy or ones that require locks.

[subway]

None of these statements are inherently true about NFS. They may be applicable to some deployments or applications of NFS, but they don't hold generally speaking.

NFS is used heavily throughout HPC for workloads that don't require a parallel filesystem (and even then, pNFS is plugging along, even if it's largely ignored in favor of Lustre et al for accessing parallel filesystems.)

[antoncohen]

It is true that there are some very high performance NFS systems out there, throughput optimized like Isilon, and some IOPS optimized like Tintri. A lot of enterprise virtualization systems store their VM disks on NFS and seem to perform decently.

But in a general sense, when I've seen suggestions to use NFS, it hasn't been the correct solution. Like a system that is designed to run on a single server, and someone wants to make it active-active HA by pointing two servers at the same data over NFS. No, that won't work. Or the high IOPS system, and someone wants to move it to a throughput optimized NFS service. Or the team the decided it was a good idea to log from multiple servers to a single file over NFS, and then complained that their log messages were not being written in order. Or having multiple web servers (HA!) serve media off a single NFS server.

In this specific case, if someone is maximizing IOPS per server, it probably isn't a good idea to use a single large volume exported via NFS and share it across them. Max IOPS for gp2 volumes is 10,000, for io1 it is 32,000. Sharing a single 10,000 IOPS volume across a bunch of servers isn't going to get you a bunch of 10,000 IOPS volumes.

Based on my experience, I instinctively question the use of NFS, it is so rarely the right solution in a cloud environment. Sometimes it is the right solution, I'm just saying it is pretty rare.

[jimktrains2]

I used to work at a supercomputing center and nfs wasn't the choice for much of anything. Lustre or AFS were the normal options depending on use case. That was a decade ago.

Also, nfs is bad at applications requiring locks. Perhaps some implementations aren't, but I haven't seen one.

[toomuchtodo]

We used pnfs at FNAL across 5000+ physical servers doing analysis of CMS detector data. Worked well.