[spookthesunset]

Don’t forget labor. You need to find talent to manage your little data center. And deal with it when it shits the bed at 4:12am on Christmas morning.

So toss in at least one SRE type person. Say $200k/year.

Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one....

Since redundancy is a thing, double that $350k. And 10pb is what they have now so double it again for 20pb. Add in $10k per rack for switches, routers, wires, etc.

So probably you are looking at a million dollars of capital plus labor to actually execute on this. And don’t forget the lead time might be a month to get the hardware and a week or two to install it. Plus all the configuration management that needs to be built up. Not to mention monitoring. So maybe a quarter of work just to have it functional.

I haven’t even factored in opportunity costs. What could this business be doing that adds more value than building out a little data center?

I dunno. Maybe it does make sense to manage your own hardware. But it helps to calculate the entire cost of ownership, not just the cost of the servers.

[Dylan16807]

> Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one....

This person's entire job is managing a few racks of hard drives? How often do you think they're actually going to get called in?

> Since redundancy is a thing, double that $350k.

True, but you can do redundancy for cheaper with parity or tape.

> And 10pb is what they have now so double it again for 20pb.

> So probably you are looking at a million dollars of capital plus labor to actually execute on this.

You can go a couple PB at a time if the upfront cost is daunting.

> Add in $10k per rack for switches, routers, wires, etc.

Yep, though that's not very much in comparison.

> Plus all the configuration management that needs to be built up. Not to mention monitoring. So maybe a quarter of work just to have it functional.

This is the one I'd really worry about.

> I haven’t even factored in opportunity costs. What could this business be doing that adds more value than building out a little data center?

You always have to keep opportunity costs in mind, but something like this can pay for itself in under a year if there's significant bandwidth cost too, and that's an amazing ROI.

[spookthesunset]

> How often do you think they're actually going to get called in?

Not often. But the server gods are a cruel mistress and it will definitely shit the bed when you are on your honeymoon, or maybe the day after your first kid is born.

[user5994461]

>>> This person's entire job is managing a few racks of hard drives? How often do you think they're actually going to get called in?

How about every day?

Quick guess how often disks need to be replaced when there are thousands of them. ;)

[Dylan16807]

You can replace disks once a month or less. That's not an on-call thing, even if you do make your $200k admin do that grunt work.

Also for one or two thousand disks I would expect less than one failure per week.

[yoz-y]

> True, but you can do redundancy for cheaper with parity or tape.

At this volumes you probably do want a carbon copy at another site to mitigate disasters like datacenter fires.

[daper]

You're right. I'm wasn't really serious. Since I'm in the middle of calculating costs of own servers in rented racks in Poland (you're right labor is more difficult than hardware) let me imagine the rest of the infrastructure (probably not all) for this "projects", just for fun:

- network switch Juniper EX4600 (10Gbps ports) + 3rd party optics ~$11k

- cheap 1Gbps switch for management access <$1k

- some router for VPN for management network - $500

- 1Gbps (not guaranteed) internet access with few IPs ~$350 / month

- 100Mbps low traffic internet access for the management/OOB network.

Time to get the hardware - 2 months. Time to rent and install hardware in rack - about 1 month. I don't count configuring the software.

This setup is full of single points of failure so I would consider it one "region" and use something like CEPH + some spare servers in each "region". That way you don't need to react immediately to hardware failures. Just send a box of hardware from time to time to the DC and use ~$20-40h/h remote hands service to replace the failed drives or whole servers. You could also buy on-site service from the hardware vendor for 1-3 years adding some cost.

I think the most important thing would be have a cleaver person who design a fault tolerant system, automatic failover, good monitoring and alerting so that any on-call and maintenance job is easy and based on procedures. That way you could outsource it. Only then it might have some sense.

[skynet-9000]

10 petabytes at AWS is $210,000 per month just for storage (even excluding AWS's very high egress and transaction pricing), so even $1M (which seems like a high estimate indeed) would be amortized in less than six months.

Also, the hardware can be depreciated, which reduces its net (of taxes) cost dramatically over time.

Five years (probably the useful life of the equipment in general) of $210,000 per month is $12.6M. That's a lot of savings.

[user5994461]

At this scale you should be able to negotiate with AWS and get a deal better than the listed price.

Regarding accounting, the AWS monthly charges are also net of taxes so it makes no difference.

[philippb]

Thanks for laying this out. Never rented rack space my self