Hacker News new | ask | show | jobs
by numbsafari 1155 days ago
I once was a customer of a DC who's roof drainage was clogged, turning it into a lake after a couple of rain storms. It then proceeded to rain inside the DC as the roof started to leak from all the pressure.

"Servers are down, I'll head over to the DC" turned into "Um... it's raining _in the DC_. Get me some tarps and get us cut over to the backup in the office".

Ah, the glory days of running out of a single co-lo across the parking lot with our "backup site" being a former broom closet.
2 comments
MisterTea 1155 days ago
As someone who has owned two commercial flat roof buildings I cant stress enough that you MUST do inspections of your roof at least twice a year. Especially if you live in a big city. I've had backups caused by kids roofing balls and bottles, stolen purse, dead squirrel, dirty balled up diapers from the neighboring apartment building. City living for ya.
link
numbsafari 1155 days ago
Yeah, I'm pretty sure in this case it was a combination of having a 4ft parapet around the entire roof, and having basically never done an inspection. Not enough drains and they were all full of leaf matter.
link
sterwill 1155 days ago
Many years ago, I managed a server room with dedicated cooling on the 4th floor of a 4-story building with a flat roof. One night the temp alarms went off, and when I showed up water was dripping off my overhead Liebert unit and onto the racks.

And it wasn't even raining outside! So I grab some plastic to cover the racks and phone in emergency portable cooling as the room's AC started failing.

It turns out earlier that day, a technician performing seasonal maintenance on a boiler tank on the roof had drained the tank and refilled it. But instead of directing the water out into a proper drain, he sent it down a convenient pipe that was actually a vent from our ceiling into the boiler house. The boiler was dozens of meters from my server room, but the water followed the old steel and plaster ceiling remnants over to my computers.

And this boiler water was more exciting than rain: it came with all the dissolved minerals, metals, and preservatives computers crave! I didn't lose any computers in the racks, but it killed the Liebert's control board.

link