[deftnerd]

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in?

It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays.

We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in battery chemistry technology that improve reliability and density, it's only a matter of time until Lithium chemistry batteries are available and that also increases the risk of fire.

Is there any reason not to move backup power to another room, or even to a separate structure like how they put backup generators on a pad outside of the building?

[mikepurvis]

For the extreme opposite of that, Google famously trolled everyone 10 years ago by announcing that every one of their servers had its own in-chassis 12V battery:

https://www.cnet.com/news/google-uncloaks-once-secret-server...

[dilyevsky]

It’s true! I wrote software that upgraded firmware on every one of those batteries without frying them up (most of the time). There was a public paper/talk on that few years back

[jeffbee]

Yes but Google also later moved to a 48VDC power architecture with the batteries at the bottom of the rack.

See http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Pl... page 6.

[tyingq]

That presentation is pretty interesting, including Google inventing their own "Switched Tank" DC-DC converters because the existing ones weren't efficient or reliable enough.

[jeffbee]

There's a whole separate paper on that:

https://storage.googleapis.com/pub-tools-public-publication-...

[LinuxBender]

And Facebook had small UPS/ATS units at the end of each row. Not sure if they still do that today it was like that when I walked through their datacenter. They did that for the purpose of power efficiency. They lost far less power by having many smaller units.

[lclarkmichalek]

What you saw likely wasn't a UPS, but an RPP

> A data center typically spans four rooms, called suites, where racks of servers are arranged in rows. Up to four MSBs provide power to each suite. In turn, each MSB supplies up to four 1.25 MW Switch Boards (SBs). From each SB, power is fed to the 190 KW Reactive Power Panels (RPPs) stationed at the end of each row of racks.

https://research.fb.com/wp-content/uploads/2016/11/dynamo_fa...

The UPS role is taken more by BBUs (Battery Backup Units). https://www.youtube.com/watch?v=KNsposM0sJE has lots of info about them.

(I work for FB, on entirely unrelated things)

[LinuxBender]

They specifically called em out as UPS and talked about the efficiency improvements over having a dedicated UPS room. I don't remember the percentage numbers though. It was orders of magnitude less power loss. It could be they ditched all of that hardware by now. This was some time ago. You might ask some of the old timers in the DC. They probably have pictures of the hardware.

[WrtCdEvrydy]

That was actually true, not an actual April Fool's day.

[tinus_hn]

It sounds rather wasteful to have one device generate mains voltage AC power, only to have another device transform it to lower voltage power and rectify it.

[xvf22]

I mean chances are they are doing DC distribution.

https://datacenterfrontier.com/google-unveils-48-volt-data-c...

[throw0101a]

> Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in?

Yes: longer cables.

See Figure 1 in the Schneider-APC white paper, where they have "Electrical Space", "Mechanical Space" (HVAC), and IT Space:

* https://download.schneider-electric.com/files?p_File_Name=VA...

Power is generated hundreds of kilometres from where it is used, so having your UPS room a few dozen metres from your actual DC room isn't a big deal. I-squared-R losses aren't going to be that huge.

Europe uses 400Y/230 for nominal low-voltage distribution (see Table 1 in above), so stringing some 400V extra copper to the PDUs, which then have 230V at the plugs, isn't a big deal.

[Reason077]

> "it's only a matter of time until Lithium chemistry batteries are available and that also increases the risk of fire."

Not all Li-ion chemistries are equal. In particular, the increasingly popular LiFePO4 (LFP) technology has much higher energy density, longer lifespan, improved environmental characteristics, and similar if not better safety characteristics compared to lead-acid.

(Besides sharing lead-acid's very low risk of fire, LiFePO4 also contains no corrosive acids which could damage and short equipment if a leak were to occur.)

[branko_d]

> lead-acid's very low risk of fire

Isn't lead-acid prone to releasing hydrogen gas?

When jump-starting a car, it is commonly recommended to connect the ground (black) cable to the chassis, not to the battery's black terminal, to avoid a spark igniting the hydrogen and causing an explosion.

[tempay]

The lead-acid batteries used in data centres are normally "sealed lead-acid batteries" and release significantly less hydrogen.

See: https://en.wikipedia.org/wiki/VRLA_battery

[doggodaddo78]

Wikipedia is wrong. VRLAs aren't SLAs, and vice-versa. Actual SLAs are very, very expensive. All VRLAs can vent, by definition.

VRLAs can overcharge and catch fire just like flooded batteries. They just don't go kaboom as much or spray acid everywhere. When having thousands of batteries in a room, one of them catching fire is inevitable. This is why batteries are in separate fire containment rooms, on nonflammable shelving, in redundant strings, and protected by FM200.

[Nextgrid]

Isn't hydrogen produced only while the battery is charging, or at worst, when discharging? If so, this still makes them much safer as they're otherwise inert while not in operation, compared to lithium-polymer which contains materials flammable at all times.

[rootusrootus]

LiPo and LFP are very different things. LiPo is probably the biggest fire risk battery chemistry in common use today. LFP is pretty safe. LTO even more so.

[Nextgrid]

Yes, I am aware of other "L" batteries that are safer. I was just commenting about how lead-acid is still relatively inert compared to the "fiery" kind of L batteries.

[tyingq]

There are plenty of datacenters with a separate battery room, sure.

[sschueller]

So far I have never been in one where this was not the case. Also they always have fire suppression.

[hinkley]

The farther your UPS is away from the server the fewer causes of power loss you can prevent.

I've seen people use UPSes to allow them to rearrange wiring. I've seen them fail by relying on the UPSes as well, of course.

If you wire the entire room with 2-3 separate electrical systems all powered off of separate remote UPSes, you can do whatever you want, but it's harder to change your mind or build out incrementally if you do.

[doggodaddo78]

Batteries burn up, especially NMC Li ion.

Battery rooms were traditionally separate, used lead acid batteries, were surrounded by thicker walls, and equipped with FM200, just like main datacenter floors. They were typically placed near the transfer and PDU switchgear. I wouldn't put anything more flammable than LiFePO4 in a battery room, much less anywhere near a server.

It's people who decide to throw away conventions, common sense, and building codes because "they know better" who get into trouble.

I suspect this datacenter company could be sued into oblivion.

[bzzzt]

I've been led around a DC in the Netherlands where they used a separate room for the UPS batteries. Not only because of safety but because the reliability of small rack-mount UPSes really sucks (and you have to match them to the power draw of the servers in the rack or pay for a lot of unused capacity)

All racks and servers were connected to dual power feeds, so even when one of the feeds goes down the servers should run fine.

[jve]

We do exactly the same you describe here in Latvia.

[walshemj]

Depends for a DC / Telco your normally feeding 48v DC from the UPS which is normally separate.

The lack of fire suppression is also very worrying.

[bcrl]

s/UPS/batteries/. DC makes uninterruptible power much more simpler than an AC UPS.

[walshemj]

Oh yes I sort of skipped over that part :-)

[bayindirh]

Our current DC has generation and UPSes in different rooms, in isolated places from each other. Both are pretty far from the actual DC itself.

[kazen44]

in most datacenters, the UPS's are stored in the same collum like structure as their generators. A Datacenter is divided into vertical columns, with eacht collum being powered by redundant power feeds, generator(s) and UPS's.