[syntheticgate]

At a certain point in EE power design you don't really want to go from 54V -> point of load for every rail (1.8V, 1.1V, 0.9V, SVI3 rails etc), so sticking with an intermediate voltage makes sense often even when viewing this from an efficiency perspective. Voltages such as 54V require different creepage and clearance requirements, so saddling every point of load regulator (of which we have many many!) with those requirements is often detrimental to an already complex board layout. Picking something like 12V or 24V as an intermediate voltage helps balance those requirements with the amount of copper you need for power delivery since the parts use low voltages but are extremely power hungry so your current at the point of load rail is a lot. This also means that your point of load regulators have to be distributed around the board near their loads otherwise the copper losses and noise would become problematic.

[hinkley]

The thermal dissipation is a function of the current squared. The heat in the conductor is a function of the size of the conductor and the surface area for heat dissipation. So these high current common rail systems you can sometimes see in youtube videos for power distribution, have great honking bars of copper in them. And in most of the videos I've seen, the video is about someone screwing one of these up, damaging the bar, and now the electrician has to wait for a new one to arrive, because they are shipped from far away and they are expensive per pound, and the dumb things weigh many pounds.

So you don't actually want the power to be at 12V for very long in a power dense rack. Their spec sheet says that each rack can pull 15KW. And that's wired for 208 or 3-phase power. That's 10 hair dryers of power per rack, so yeah maybe you shouldn't step it down until the last responsible moment.

Do any parts of the rack run at the full 54V? That would make for some very nice cooling fans.

[syntheticgate]

In Oxide's design, we do have a 54V DC busbar so that's what the rectifiers put out, and runs vertically up and down the back of the rack. The power connection into each of the cubbies for the sleds, and the power into the sidecar switches connect to this bus bar at 54V. Each of these assemblies has an intermediate bus converter IBC that does the 54->12V conversion on board and 12 and other lower rails are used for the various supplies required.

We do run the 54V to our fans (in both sleds and switches) without additional DC-DC conversion as those can be fairly power hungry and we can buy reliable fans that are rated for this voltage.

Our sleds actually don't connect directly to the bus bar to help mitigate some of the "oops" factor as they're going to be potentially mated and unmated buy customers as they reconfigure, upgrade or support things. The sled cubbies are wired to the bus-bar and support hot insertion of the sleds. And yes, while possible, an in-field replacement of the bus bar wouldn't be fun, but in our design it's a big copper bar hidden away so the risk of damage or dropping stuff into is minimal.

So our 12V IBC design gets us into more normal range for commodity point-of-load supplies, and balances the losses due to higher current at 12V vs the complexity of dealing with 54V all over the boards. For the AMD parts, we also have to have supplies that deal with SVI2 or SVI3 where the part itself can adjust its voltage at run-time for efficiency. These are pretty complicated devices (like the RAA228218) that we're happy to not have to design ourselves and they have expected operating envelopes for their supply-side rails that don't work at 54V.

[hinkley]

One of the things I recall from automotive discussing the switch to 48v systems as a route to hybrid vehicles, is that the coils in motors and generators can be smaller when the voltage is higher. An example they gave is that the alternator could drop about a pound of copper out of its windings. Do the 54v fans have smaller hubs?

[collinmanderson]

Does Oxide require 3-phase power? Do datacenters typically provide 3-phase power?

[hinkley]

Their spec sheet shows 208 and 3 phase as options. 3 phase is smaller wiring, and with 15KW per rack I could see how that would quickly become a problem.

[jeffbee]

It's certainly the current mainstream style to have an intermediate voltage rail of 12V or more. But this OCP talk from a few years ago was interesting, showing a prototype direct 48V-1V conversion with high efficiency.

https://www.youtube.com/watch?v=JQHiKIfrwI0

[syntheticgate]

Yes it certainly can be done, but there's a cost and design complexity with doing that too. I did a quick count of gimlet (our server sled's) power rails and got to over 26 different power domains, and I probably missed a few in my quick scan! It's unclear to me if the efficiency gains from re-doing these with something more exotic to go from 54V would make enough of a difference to justify doing so, and we'd still end up with some stuff like the SVI2/3 controllers needing an intermediate rail (or have to go design those ourselves too) and some analog rails needing LDOs for noise rejection reasons etc. As mentioned before, creepage and clearance at higher voltages cascades into layout complexity and pain if you have to run it everywhere on the board: but for the same reasons we're talking about this, we can't very well do a 54V:1V conversion in the back of the sled and run it all the way to the front- losses, noise etc.

As with all things engineering is a series of tradeoffs and right now, an IBC from 54V to 12V has been a reasonable design point for us.