Wasabi is a good option. They’re S3 compatible and don’t charge any egress or ingress fees. Been using them for a few years. Great speeds and customer support.
To be fair, purchasing and hosting even the most basic mirrored RAID array of that scale comes to well over half a million for the disks alone. Then you need to manage them.
10x Supermicro SSG-6049P-E1CR60H servers (60 x 3,5" HDD in 4U enclosure) - $5k each
600x WESTERN DIGITAL Ultrastar DC HC550 18TB (10800PB in total) - $500 each
~$350k in hardware, up to 20kW energy consumption, should fit in two rack towers. You can host it for about $1.5k somewhere. All assuming no redundancy :)
Don’t forget labor. You need to find talent to manage your little data center. And deal with it when it shits the bed at 4:12am on Christmas morning.
So toss in at least one SRE type person. Say $200k/year.
Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one....
Since redundancy is a thing, double that $350k. And 10pb is what they have now so double it again for 20pb. Add in $10k per rack for switches, routers, wires, etc.
So probably you are looking at a million dollars of capital plus labor to actually execute on this. And don’t forget the lead time might be a month to get the hardware and a week or two to install it. Plus all the configuration management that needs to be built up. Not to mention monitoring. So maybe a quarter of work just to have it functional.
I haven’t even factored in opportunity costs. What could this business be doing that adds more value than building out a little data center?
I dunno. Maybe it does make sense to manage your own hardware. But it helps to calculate the entire cost of ownership, not just the cost of the servers.
> Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one....
This person's entire job is managing a few racks of hard drives? How often do you think they're actually going to get called in?
> Since redundancy is a thing, double that $350k.
True, but you can do redundancy for cheaper with parity or tape.
> And 10pb is what they have now so double it again for 20pb.
> So probably you are looking at a million dollars of capital plus labor to actually execute on this.
You can go a couple PB at a time if the upfront cost is daunting.
> Add in $10k per rack for switches, routers, wires, etc.
Yep, though that's not very much in comparison.
> Plus all the configuration management that needs to be built up. Not to mention monitoring. So maybe a quarter of work just to have it functional.
This is the one I'd really worry about.
> I haven’t even factored in opportunity costs. What could this business be doing that adds more value than building out a little data center?
You always have to keep opportunity costs in mind, but something like this can pay for itself in under a year if there's significant bandwidth cost too, and that's an amazing ROI.
> How often do you think they're actually going to get called in?
Not often. But the server gods are a cruel mistress and it will definitely shit the bed when you are on your honeymoon, or maybe the day after your first kid is born.
You're right. I'm wasn't really serious. Since I'm in the middle of calculating costs of own servers in rented racks in Poland (you're right labor is more difficult than hardware) let me imagine the rest of the infrastructure (probably not all) for this "projects", just for fun:
- some router for VPN for management network - $500
- 1Gbps (not guaranteed) internet access with few IPs ~$350 / month
- 100Mbps low traffic internet access for the management/OOB network.
Time to get the hardware - 2 months. Time to rent and install hardware in rack - about 1 month. I don't count configuring the software.
This setup is full of single points of failure so I would consider it one "region" and use something like CEPH + some spare servers in each "region". That way you don't need to react immediately to hardware failures. Just send a box of hardware from time to time to the DC and use ~$20-40h/h remote hands service to replace the failed drives or whole servers. You could also buy on-site service from the hardware vendor for 1-3 years adding some cost.
I think the most important thing would be have a cleaver person who design a fault tolerant system, automatic failover, good monitoring and alerting so that any on-call and maintenance job is easy and based on procedures. That way you could outsource it. Only then it might have some sense.
10 petabytes at AWS is $210,000 per month just for storage (even excluding AWS's very high egress and transaction pricing), so even $1M (which seems like a high estimate indeed) would be amortized in less than six months.
Also, the hardware can be depreciated, which reduces its net (of taxes) cost dramatically over time.
Five years (probably the useful life of the equipment in general) of $210,000 per month is $12.6M. That's a lot of savings.
I see from their home page they do not charge for egress, but the FAQ clarifies this is only valid if your monthly egress total is less than or equal to your storage total, otherwise they suspend your service. Should be clarified on the home page in my opinion. At least with an asterisk beside "No egress charges".
edit: perhaps their RCS option would be cheaper if you know exactly how much data you need to store in advance.