[shiftpgdn]

Kind of a fun ancedote: In "the old days" when you'd run a server room in your office we had a very large HPC cluster + a significant amount of storage and other one off servers on the top floor of a mid rise office building. I eventually moved it all to a former nuclear fallout facility where our systems were three floors down and under a gigantic pool of water. Error rates and random crashes fell off IMMEDIATELY. I believe Microsoft reported similar findings with their submerged data center.

[sandworm101]

Wonderful. Yet another reason for the IT office to be relegated to a sub-basement.

[yunohn]

Why the whole IT office? Just the hardware running the software, not the humans who develop it.

[sandworm101]

Most offices/businesses have nothing to do with software development. Most IT shop are in basements, or at least on lower levels near the core servers and other infrastructure they manage. "The IT Crowd" was a stereotype but was still based on the reality of most businesses. I have never seen or even heard of an IT shop with a skylight. Even windows are rare.

[wmichelin]

I feel like OP was joking

[sunsipples]

should turn the entire floor into a big toilet

[sandworm101]

There are places lower than the basement.

[kevin_nisbet]

Yea, when looking into these myself I did float in conversation that we'd need build a giant pool over our data centers. It was more of a joke though and never went anywhere.

Although our error rate was more like 1-2 per week on the equipment I was looking at.

[sandworm101]

Pools are expensive. The same weight of gravel or sand would produce a similar effect. Put the server farm under a multi-level car park structure.

[shiftpgdn]

Water is one of the most effective radiation shields we have for cosmic rays. NASA has contemplated using a water shield for Mars astronauts. There is a reason most fallout shelters are built under a pool when possible.

[sandworm101]

Nasa has also contemplated using human feces as shielding. The mixed use, dual purpose systems for spaceflight are a special case. What really matters is mass. Sand is cheaper than pool water.

[kevin_nisbet]

Doing some quick searching, looks like there has been some research into what materials are better than others for shielding for cosmic rays. The couple minute look to me suggests there is more complexity than just mass, different materials have different properties, including at different energy levels for the cosmic rays. Although I suppose material cost and construction cost would also be a factor, where alot of sand might be easier to build a facility than the equivalent shielding in water. Although we're probably at the point where it's better to use something like a mine that is no longer used.

[wang_li]

Sun’s UltraSPARC II CPU modules were claimed to be particularly susceptible to cosmic rays circa 2000.

[phonon]

https://www.forbes.com/forbes/2000/1113/6613068a.html?sh=28e...

https://www.computerworld.com/article/2584471/q-a--mcnealy-d...

http://www.cedix.de/VorlesMirror/Band1/UnsafeAtAnySpeed.pdf

[p_l]

Not so much susceptible as "came with their own radioactive source"

[digikata]

A reduction in floor and building vibration might have contributed too?

[oconnor663]

Or for example, old cables and improperly installed cooling fans might tend to get fixed during a move. Hard to know for sure.

[shiftpgdn]

Well yeah, that's why it's presented as an anecdote. :)

[oconnor663]

Totally. Kind of reminds me of http://www.ibiblio.org/harris/500milemail.html

[digikata]

I was recalling this paper characterizing loss of system drive performance due to environmental vibrations (though I'm sure there are others).

https://www.usenix.org/legacy/events/sustainit10/tech/full_p...

[suifbwish]

Awesome. Next time clients call about their services being down I am totally using this one.

[hinkley]

I've had a couple outages just from people jiggling networking cables, back when nobody cared about locking them down.

I wonder how much of that was cosmic rays and how much was just less foot traffic resulting in fewer errors.

[TTPrograms]

Can you give OOM of any of the improvements? Eg. crash rates halved?

[brobinson]

I have no experience with tracking this kind of thing. How do you do it, what kind of analytics/tracking program is used, etc.?

[lolthishuman]

And people still think the sun is nuclear. Ha!