[rolph]

OK, Hanlon's razor applies here:

https://en.wikipedia.org/wiki/Hanlon%27s_razor

recall Stuxnet, its possible this was an attack, somesort of malicious mod to firmware, but_ Hanlon's razor.

from the reddit:

"throwawayfordays75 1399 points 8 hours ago*2

Throwaway since I have first hand knowledge. Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the network or compute equipment and some failovers did not work as expected. "

At this point im wondering what "utility work" was happening.

[hinkley]

That user posted an update:

"everything minus core network gear was manually being unplugged from any PDUs to help the control the initial power-on."

Can't we... Why don't we have rack hardware that can handle this situation? I thought some HDD RAID solutions had circuitry to keep them from browning themselves out while spinning up the disks. I guess I'm surprised this isn't a solved problem at the rack level now.

Or have we been so focused on never cold booting a rack of servers that we haven't spent any effort on foolproofing of cold booting a rack of servers?

[Edit: answering my own question] apparently these exist and are called Managed PDUs. Can we deduce WF doesn't have them?

[mustardo]

Some servers "stage" power on to disks. as spinning up disks is often the largest source of inrush current for e.g. a 10 disk server might only apply power to bays 2 at a time with a delay between each, this doesn't solve the inrush current problem of switching on a rack full or servers together (where managed PDUs come in)

[rolph]

it would seem to be a possibility. im also thinking 10 finger power up sequence, as in: insert power cord 1 into powersocket 1, wait till audible beep, insert power cord 2 into power socket 2, wait till audible beep etc. :-D