[th5]

Is it not worrisome that they only have one db server with no hot-backup or fail-over db machine? I suppose many components on that one machine are redundant - disks, cpu's... but theres got to be many points of failure in there as well right?

[nick_craver]

Both NY-DB01 (runs everything but Stack Overflow) and NY-DB03 (runs Stack Overflow) have identical backup counterparts: NY-DB02 and NY-DB04. NY-DB04 is on a mirrored config and is always a few minutes behind, while NY-DB02 is restoring scheduled backups. With SQL Server 2012, the backup/mirror is greatly improved and both boxes will have fully-hot spares in a replica configuration. In 2008 R2, SQL Server just doesn't handle mirroring 100+ databases well.

[griffordson]

And those are all in the same data center, right? How much downtime do you guys have a year?

[nick_craver]

The hot backup is in the same New York datacenter, yes. We also have daily backups across the country in our original Oregon datacenter. The whole OR setup is getting love to be a much more resilient failover location as we speak (that'll be the topic of my next post). As for downtimes, pingdom says we were down 7h 6m 23s last year, so 99.92% uptime (note: not nearly all of that was DB related).