[avleenvig]

RO filesystems can be bad, but usually they're soft failures for us: * Memcache can still work just fine * Db servers stop responding (and the app handles that fairly gracefully) * Web servers serve files from a RAM disk, so they keep working

No reason against 10G copper specifically - we haven't had to address the problem in detail yet. When we do, we might choose copper. Depends what happens when it happens :-)

We had a very nasty incident a few months ago, where the drive in one of the LDAP servers died. Well, it sort-of died. It started to time out a lot but didn't go fully offline. openldap kept running, but when you connected to it, the TCP connection would open and hang. This meant that all of our servers saw the server as "OK", but LDAP stopped working and caused all kinds of brilliant havoc :-)

[donavanm]

Interesting. I was thinking failure to persist updates, inconsistent health check results, those types of nasty corner cases.

Sure. Premature optimization and all.

Huh. SSD from a major chip vendor perhaps? I had one of those bite me just a few weeks ago. in my case it would have been fine if the drive offlined itself. but the intermittent io pauses killed performance, but not health checks. Actually, same type of problem RO FSs make me scared of.