[rtkwe]

They're pretty specific to the datacenter use case with no outputs and they need to be cooled externally, principally through the very loud high speed fans used in data centers. I suppose you could strap a fan to one and put it in a normal case or maybe make a dedicated after market cooler (like the water blocks made for water cooling cases).

[jtolmar]

I think there's a market for a home AI server that can run an LLM or video gen model behind a web frontend. Not literally a raspberry pi strapped to an H100, but something with lopsided enough specs that people joke it is.

And precisely because it's such a huge headache to do yourself, I think a small company could make a nice business wrapping up used datacenter cards in that sort of server.

[everforward]

I'm doubtful they can, based on current inference prices. An H100 draws about 50W while idling, which is ~$8/month at average US electricity prices. They also draw ~400W while active.

The electricity prices are relevant because if you paid $0 for your H100 and didn't use it a single time, you could buy millions of tokens in inference just on the electricity it draws while idling. If you can't keep that thing saturated through the night, you're probably underwater overnight. Likewise, it's too small to run even the frontier open source models so you need to be able to live with worse models.

Max power matters because you aren't going to run many of those H100s before you blow breakers in most houses. Newer houses in the US are 15A service to non-kitchen breakers, so 1650W (that might be peak rather than continuous, not sure). If you're plugging that into an existing run, you could maybe run 2 before you start blowing breakers? You can't just plug 4 H100s into the wall in a normal house.

Maybe I'm wrong, though. I'd be curious, it'd be neat to run my own inference for something more than what'll run on a 3080.

[smalltorch]

Most houses have a 200amp panel and extra space for larger breakers.

A electrician can plop in a electric car charger for instance, that is a 40-50 amp circuit.

[everforward]

The median home age ranges from 26 to 63 depending on state, and 100 amp service was common into the 90s. 100A service is probably more common than you think, I'd bet we teeter around them being equally common.

My home still has 100 amp service (as does almost the whole city, most of it is a historic district), but it was built in 1910 so that's not shocking.

> A electrician can plop in a electric car charger for instance, that is a 40-50 amp circuit.

Oh yeah, I was mostly talking about having to upgrade the service to the house from the pole. A new circuit with a single outlet isn't horribly expensive, but needing to upgrade service is pretty rough.

[rtkwe]

Newer houses yes probably have the overhead. We put in a charger for my wife (because she drives 100 miles a day we needed a pretty beefy level 2) and had to replace our whole panel because 1) we had 125A so failed the load calc and 2) we had the infamous Zinsco panels and the electricians refused to squeeze one in even if we did pass the load calculations. Luckily we had a basement so they only had to extend the circuits a little instead of rerunning them all over the house.

[HWR_14]

I'd imagine that you can have the power to the board shut off when not in use.

But I never assumed it was to be cost competitive at current token rates. I assumed it was for the same reasons people might use open source hardware. Freedom to tinker, etc.

[rtkwe]

Basically sounds like a hobo version of PCI-e hotswapping which is a thing on enterprise motherboards but not many consumer boards. Also depends on if the board can actually power off the slots which I'm not sure many are setup to do.

[trollbridge]

A typical oven plug can offer 9.6kW (so your 24-node H100 can keep your kitchen nice and warm). My furnace was 16.5kW, which once failed in an odd way and was stuck on for about a week. The people living in the house simply opened the windows until I got around to fixing it. I estimate it wasted $400 on the power bill.

[Bombthecat]

Meh

Models are too big now

[timmmmmmay]

This is also true of the older P100 and hobbyists do this stuff now, today

[chasd00]

> Very loud high speed fans

if you haven't heard a 5u server intended for a datacenter rack come to life it's quite the experience. Sounds like a plane taking off.

[gerdesj]

"Sounds like a plane taking off."

Helicopter. I live in Yeovil, Somerset, UK - there's a helicopter factory just down the road. I had a IBM "AS/400" or whatever they are called now in our computer room rack for a customer and it made nearly as much noise as everything else put together. It was clearly tuned for start up noise to impress because they would fire up in sequence, rise to a crescendo and then slow down in sequence to just a din instead of painfully loud.

A switch or PC server on boot will normally run up cooling fans instantly to max as a default protection mechanism until the "OS" has started and sensors read and then the fans will slow down to deal with the actual thermal load.

If you switch off your air conn, it gets noisy, quickly. Recently in the UK we are seeing routine temperatures around 30C and we broke 200 odd year records for temperatures a few weeks back. I know its even worse elsewhere but our infrastructure is not designed for this. Here we are at the same latitude as Calgary AB!

[LargoLasskhyfv]

That's called staggered spin-up. One does that for sets of HDDs, too.

Anyway, what is it good for? To not overload the PSU for one, because spin-up needs more energy, until it has reached it's intended range of RPM. (for fans & HDDs)

Another reason to spin-up high initially, to fall down to slower when up, is to overcome friction in the bearings, to get stuck things unstuck, and shake dust off. (for fans)

[gerdesj]

Tell that to my switches and PC servers. They all spin up their fans in one go as soon as power good is received and the initial BIOS thingie has stubbed out its fag on the floor.

The PSUs are painting their nails when the fans go off initially. The GPUs, RAID controllers and co are still waking up.

A further counter example is an Equallogic PS6500. That has 48 3.5" spindles in it and three PSUs plus a few fans and two controllers. The whole thing boots in a couple of minutes.

[rbanffy]

My workstations are usually tower servers, which are the same design as their 4 and 5u rack counterparts.

Until the thermal management kicks in, they sound like jet planes. When the thermal management starts and assesses the required cooling, it’ll throttle down the fans to reasonable levels.

That is, until the moment you push the machine to its limits. When then happens, you might get back to the same levels of the boot time, but it’ll require you to push everything to the max - CPU, memory, storage (all 24 bays) and so on. For a normal user, there is a lot of room and it’s virtually impossible, even with a dozen of Teams windows open.

[baby_souffle]

Yes, but the line between "reasonably quiet" and 100% isn't a step function.

If these do end up in Home Labs, it's going to be a server rack in the basement and not the server rack at the other end of the room.

[rbanffy]

> If these do end up in Home Labs, it's going to be a server rack in the basement and not the server rack at the other end of the room.

If they are too loud, yes. In my part of the world basements are not common, but my shed would be a sensible place for one.

[CamperBob2]

Closest thing I can think of is the old LucasFilm "THX" trailer. Only there's no release segment, not until the BMC comes up a minute or so later.

[gessha]

If you check eBay for V100 SXM variant, you will often see them sold in combination with a cold plate, together with a PCIe carrier board. There’s definitely a market for them.

[lelanthran]

Meh.

You don't need much of a fan if you are prepared to pump a lot of water.

I mean, sure, if you're trying to cool the thing with 100ml of water, then yeah, you need a fan.

OTOH I have an unused car radiator in my garage, a quiet and cheap pump, and space to hang that radiator outside the window. It's barely an afternoon's worth of work if the H100s already have liquid-cooling intakes/exhausts on them already (I dunno, I have never seen one in the flesh).

I'm pretty certain 10+ litres of constantly circulating water (antifreeze, in my case) through a car radiator would be sufficient to cool down 2x H100s. Hell, add in 10 RPM used car cooling fan and I can probably cool 20x of those running at full-bore.

If anyone wants to ship me a bunch of H100s, I'll happily build it all out, take pics and videos and post it back here. Just sayin' :-)

[bayindirh]

Or you can secure yourself a DLC one and try to feed it with the correct regime (liquid composition, temperature and flow rate). I'd say good luck.

These things get hot and are fussy about their requirements.