Hacker News new | ask | show | jobs
by hinkley 8 days ago
My ex was late from work once a week because the company did commercial real estate logistics (sort of similar domain here) and she had the job of going to the secure data center and grabbing a backup disk out of the cage and transferring it to a safety deposit box.

The dumb thing was the bank was two blocks from the data center and less than eight (six?) from the office so catastrophic events might have hit both or all three. The owner kept a second copy at his house, and that was the only geographically separated copy.

1 comments

Yes, quite, this.

One of the lessons (and subsequent data integrition / continuity of business practices) learned from the 9/11 attacks in New York City, and destruction of both primary and secondary data stores of multiple entities (as well as several emergency-response agencies at various government levels from city to federal) was that essential data and operational roles need to be redundant across very widely-separated locations.

E.g.: from "Discussion note prepared by staffs of the Federal Reserve, the New York State Banking Department, the Office of the Comptroller of the Currency, and the Securities and Exchange Commission, for discussion at a meeting on February 26, 2002 at the Federal Reserve Bank of New York":

[I]t was clear that business continuity planning had not fully taken into account the potential for wide-area disasters and for major loss or inaccessibility of critical staff. Contingency planning at many institutions generally focused on problems with a single building or system. Some firms arranged for their backup facilities to be in nearby buildings on the assumption that, for example, a fire might incapacitate or destroy a single facility. Very few planned for an emergency disrupting an entire business district, city, or region. As a result, some firms lost access to both their primary and backup facilities in the aftermath of the September 11 events, severely disrupting their operations.

"Summary of "Lessons Learned" from Events of September 11 and Implications for Business Continuity" February 13, 2002" <https://www.sec.gov/divisions/marketreg/lessonslearned.htm>

Geographic distances were rarely considered prior to 9/11. Most companies were comfortable replicating data intercampus or to a facility within a few miles of a primary data center. A few firms, such as Nasdaq, actually replicated data out of state.

"9/11: Top lessons learned for disaster recovery" (Sep 9, 2011) <https://www.computerworld.com/article/1545389/9-11-top-lesso...>

Also: "Hard-Earned Disaster Recovery Lessons From 9/11 and other disasters" (2025) <https://backupcentral.com/hard-earned-disaster-recovery-less...>

Terrorist attacks, natural disasters, widespread power and other failures, etc., can all have impacts across many kilometres, possibly many hundreds. Redundancy equates to survivability here.

More broadly, information and data services are fundamentally about risk management and disaster response, as realised during 9/11 (as well as multiple earlier and subsequent incidents) in ways which weren't fully realised at the turn of the millennium. Or even today in some organisations.

Make backups. Test backups. Practice switchovers between primary and secondary (or multiple redundant) operations. And keep those resources and facilities widely separated.

Odd that a terrorist attack achieved what historical knowledge of flood and earthquake damage did not.
Earthquakes and floods tend not to specifically target major banking and information centres. Terrorist attacks have intentionality.

That said: San Francisco / Silicon Valley are home to both tech and an active seismic zone, and there's long been a tacit understanding that data centres are best located (or at least backed up) at quite some distance. One large brokerage firm located at the time in downtown SF had its datacentre in Arizona, and it wasn't the only such instance. Many other organisations opted for nearer-but-still-remote locations in the Central Valley.

I've been out of direct involvement in the space for a while, but my understanding that Arizona nad parts of Nevada were probably the safest locations with respect to natural disasters that could affect data centers.
Absent heatwaves and water shortages, generally, yes.

Pros: Cheap land, generally predictable weather, few severe storms, little seismic activity, reasonably close to population centres (Phoenix, Tuscon, Las Vegas).

Cons: Little water, wildfire hazard, heat, sand and dust storms.

Inland northern locations (Dakotas, say) might also work. Moderate tornado risk, cooler weather (particularly in winter). Subject to winter storms, but generally more water availability. Technical populations might be difficult to recruit.

Water is definitely an issue--I thought it and didn't write it down. The heat, yes, but also a dry heat :-) so some evaporative cooking; I think Intel has at least one fab plant in Arizona. As I say, no longer really in the space, but areas of the northern Midwest not subject to flooding are possibilities. For datacenters, onsite technical staff is not really an issue as datacenters have very little of that after initial construction.
That was why they wanted to store nuclear waste in a hole in a mine in Nevada, but then someone found a fault line that we didn't know much about.
And a House Speaker.

Yucca Mountain is quite probably more than adequate to task. But nuclear waste's long-lived nature makes long-term planning problematic. Human institutions such as corporations, not-for-profits, governments, and even language, writing, and culture, are minuscule by comparison.

Consider that the world's worst nuclear accident site has seen four changes in national government (USSR -> Commonwealth of Independent States -> Russian Federation -> Ukraine), and is presently enemy occupied territory in a free-fire active war zone.

There have also (rarely) been intra-fault earthquakes. A fairly large one in Kansas if I recall correctly.