[mattnewton]

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

[J_Shelby_J]

I’m running two 3090s on a 700w psu. You definitely can get more than that out of 2000w bus.

I wrote a blog on reducing the power limits of nvidia gpus. Definitely try it out. https://shelbyjenkins.github.io/blog/power-limit-nvidia-linu...

[XMasterrrr]

Hey man, I think I came across your blog at some point before while trying to figure out my own power plan for this beast (check my comment to OP for more context), so kudos to you for that.

I would say that power limiting is a potential work around, and it should work perfectly fine for inference, but when it comes to trainning you will want to squeeze every ounce of power. So, depends on your goal.

What CPU/Mobo/Storage are you running with those two 3090s for a 700w to work? I am gonna say, if at any point you're pushing more than 500w out of that PSU, you might be risking the 80% safety rule. I would have at least used a 850w just to be safe with two 3090s + rest of hardware.

[smcnally]

Thank you for this post. I’d read it in ~June and it helped quite a bit with manual ‘nvidia-smi’ runs. I just recently created the systemd service description and am still delving related power and performance possibilities.

[tcdent]

I can't believe a group of engineers are so afraid of residential power.

It is not expensive, nor is it highly technical. It's not like we're factoring in latency and crosstalk...

Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise.

[gizmo686]

A good engineer knows the difference between safe and dangerous. Setting up an AI computer is safe. Maybe you trip a circut. Maybe you interfere with something else running on your hobby computer. But nothing bad can really happen.

Residential electrical is dangerous. Maybe you electrocute yourself. Maybe you cause a fire 5 years down the line. Maybe you cause a fire for the next owner because you didn't know to protect the wire with a metal plate so they drill into it.

Having said that, 2 4090s will run you aroud $5,000, not counting any of the surrounding system. At that cost point, hireing an electritian would not be that big of an expense relativly speaking.

Also, if you are at the point where you need to add a circut for power, you might need to seriously consider cooling, which could potentially be another side quest.

[XMasterrrr]

I agree with you on all of that. I went down the rabbit hole to understand what's up, but I also hired someone and told them exactly what I wanted: breakers amps and volts, outlets type, surge protector over the entire breaker box up to 120k, etc (I am going to be writing about power and electricity in part 3 of this blogpost series). Electricity was on top of the things I was not going to cheap out on because the risk vs reward made no sense to me.

Re: cooling; I have an AC vent directed on the setup, plus planned out in-out in the most optimal way possible to maximize cooling. I have installed like 20 more fans since taking these pictures :D

[vunderba]

Just a slight clarification, an RTX 4090 card currently runs about $1700 USD at least in the states, so it's more like $3500 pre-tax.

[XMasterrrr]

A year ago it was a struggle to get one for anything below $2100 pre-tex. Glad it came down a bit.

[tjoff]

Add to that is that it is likely illegal to do yourself. Which of course has implications for insurance etc.

[m-s-y]

In the US, it’s fully legal to perform electric/plumbing/whatever work on your own home.

If you screw it up and need to file a claim, insurance can’t deny the claim based solely on the fact that you performed the work yourself, even if you’re not a certified electrician/plumber/whatever.

What you don't want to do is have an unlicensed friend work on your home, and vice versa. There are no legal protections, and the insurance companies absolutely will go after you/your friend for damages.

Edit: sorry this applies to owned property, not if you’re renting

[earleybird]

In my jurisdiction I can certainly do the work but am under the same requirements to pull a permit and pass a provincial inspection. It very quickly becomes the most effective to have an electrician involved, maybe not for all the work but some of it. They're more that willing to review the work you do and talk about it. Think of it as pair coding - great opportunity to learn and they'll tell you when you've done a good job. (at least the ones I've found)

[gizmo686]

Around here, the bar is lower for work on your own property, but you still need to be qualified by the county to be allowed to do so. Qualification consists of a 2 hour open book exam, where the book is a copy of the national electrical codes.

https://www.montgomerycountymd.gov/DPS/Process/combuild/home...

Granted, if you actually do unlicensed work in your house, no one will know. But it is still illegal.

[jefftk]

Depends on the state and municipality. Mine doesn't allow homeowners to pull electrical permits.

[scubbo]

Extraordinary claims require extraordinary evidence.

[zamadatix]

As with most regulations in the "US" I have a feeling the answer is really something like "Depending on the city and state you live in the answer lies somewhere between 'go nuts' and 'that could lead to criminal charges and you being liable for everything that happens to the house and your neighbors kitchen sink'".

[tourmalinetaco]

It‘s hardly an extraordinary claim. Just because you can’t install a ceiling fan doesn’t mean it‘s an “extraordinary” feat that is “likely illegal”.

[bigiain]

> Also, if you are at the point where you need to add a circut for power, you might need to seriously consider cooling, which could potentially be another side quest.

There should be an easy/reliable way to channel "waste heat" from something like this to your hot water system.

Actually, 4 or 5 kW continuous is a lot more than most domestic hot water services need. So in my usual manner of overcomplicating simple ideas, now I want to use the waste heat to run a boiler driving a steam engine, perhaps to directly mechanically run your air conditioning or heat pump compressor.

[raxxorraxor]

Instant water heaters use up to and sometimes even more than 27kW. Of course boilers use less, but still...

These aren't power requirements that are insurmountable. They would get pricey though and I wish my rig for computing would use something around .1kW under load...

Using the heat from PCs would be nice. I guess most just use them as electrical heaters right now.

[XMasterrrr]

Let me know if you figure it out, I would be really interested hahaha

[BizarroLand]

I'm doing this myself now. I have a homelab server setup and a hybrid water heater.

Stuffed the homelab next to the air intake of the water heater, now when I need hot water my water heater sucks the heat out of the air and puts it into the water.

It's obviously not 100% efficient, but at least it recaptures some of the waste heat and decreases my electrical bill somewhat.

[lolinder]

> I can't believe a group of engineers are so afraid of residential power. ... Read a quick howto, cruise into Home Depot and grab some legos off the shelf. Far easier to figure out than executing "hello world" without domain expertise.

The instinct to not touch something that you don't yet deeply understand is very much an engineer's instinct. Any engineer worthy of the title has often spent weeks carefully designing a system to take care of the hundreds of edge cases that weren't apparent at a quick glance. Once you've done that once (much less dozens of times) you have a healthy respect for the complexity that usually lurks below the surface, and you're loathe to confidently insert yourself confidently into an unfamiliar domain that has a whole engineering discipline dedicated to it. You understand that those engineers are employed full time for a reason.

The attitude you describe is one that's useful in a lot of cases and may even be correct for this particular application (though I'm personally leery of it), but if confidently injecting yourself into territory you don't know well is what being an "engineer" means to you, that's a sad commentary on the state of software engineering today.

[tcdent]

Sir, this is "Hacker News".

[lolinder]

So did you mean "I can't believe a group of hackers are so afraid of residential power"?

[3eb7988a1663]

People can and do die from misuses of electricity. Not a move-fast-and-break things kind of domain.

[varispeed]

You only "break" once...

[lbotos]

I've been learning Japanese and a favorite of mine is: 一体

Which is used as "what the heck" but it's direct kanji translation is one body.

https://jisho.org/word/%E4%B8%80%E4%BD%93

[varispeed]

Fun fact as a kid I stuck my fingers in the loose mains wires as we were playing at unfinished building. The wires were live and I still remember it felt like it's going to break my arm. Fortunately I only got a slight burn. This got me interested in electronics which I started studying later in my life.

[fhdsgbbcaA]

You’re forgetting many people have landlords who aren’t exactly keen on tenants doing diy electrical work.

[tourmalinetaco]

I’m hardly surprised, this is primarily a programming discussion website, and the highest voltage read on an average day here is in mV. It’s natural to be leery of things you have no experience in.

[fragmede]

your car is 12 volts, and USB is 5 volts; 12 or up to 20 these days for laptop charging. My computer's CPU is probably 1.8 volts but I can't remember the last time I had my multimeter on that, but that's still more than millivolts.

[raxxorraxor]

Probably meant milliampere, specifically 1 milliampere. But yes, usually lightweight engineers are familiar with TTL and limit themselves to 5V. 12V+ is another arcane realm you don't want to touch.

Some old serial ports had 12V and a high max current. The DIY things you attached here were prone to kill your mainboard.

Voltage/current is either 0 or 1. Anything higher kills software developers instantly.

[matt-p]

In telecoms 48V dc is very common and not always even connectorised! It's "safe-ish" but DC makes me more nervous than 240v, big thick 400A cables into a rack are quite intimidating to see but the main issue is DC is sticky and doesn't have the safety protections of RCDs etc. Indeed you are lucky to get a working isolator.

[littlestymaar]

That's technically correct, but irrelevant: you cannot kill yourself with 12 or 20V any more than with 10mV. 120V or 230V is another story.

That being said, it's still very easy not to kill yourself with 120/230V: just shut down the power before touching anything.

[wpietri]

Ah yes, the "move fast and burn your house down" school of "engineering".

[phil21]

Adding a bog-standard breaker and a short conduit run is about as simple as it gets for electric work. It’s rather low risk if you simply read the code and follow it.

If you know nothing about basic electric work or principles, sure - spend the $500 to have an electrician add a 30 or 50A 220V outlet near your electric service panel. Totally reasonable to do as it is indeed dangerous to touch things you don’t understand.

It’s far less complex and less dangerous than adding an EV charge point to your garage which seems to be quite common for this crowd. This is the same (less, since you typically have a lot more flexibility on where to locate the outlet and likely don’t need to pull through walls) complexity as adding a drop for an electric stove.

Where the “home electric hackers” typically tend to get in trouble is doing stuff like adding their own generator connection points and not properly doing interlocks and all that fun stuff.

If you can replace your own light switches and wall receptacles you are just one step away from adding an additional branch circuit. Lots of good learning material out there on the subject these days as well!

[wpietri]

I'm not saying people shouldn't add breakers. I'm saying your talking like people are scaredy-cats and comparing it to working with toys or hello world is exactly the kind of of macho nonsense that leads people to do shoddy engineering.

As a hobby, I restore pinball machines. A modern one is extremely careful about how it uses power, limiting wall current to a small, normally-sealed section of the machine. And even so, it automatically disables the lower-voltage internals the moment you open the coin door. A 1960s machine, by contrast, may not have a ground at all. It may have an unpolarized plug, and it will run wall current all over the place, including the coin door, one of the flippers, and a mess of relays.

In the pinball community, you'll find two basic attitudes toward this. One is people treating electrical safety about as seriously as the people who design the modern machines. The others is people who think anybody who worries about a little wall current are all pussies who don't have the balls to work on anything and should just man up and not worry about a little 120V jolt.

The truth is that most people here are not engineers of any sort. We're software developers. We're used to working in situations where safety and rigor basically don't matter, where you have to just cowboy ahead and try shit. And that's fine, because control-z is right there. I've met people who bring that attitude to household electrical work, and they're fucking dangerous. I know one guy, quite a smart one, who did a lot of his own electrical work based on manliness and arrogance, and once the inspector caught up with him, he immediately pulled the guy's meter and wouldn't let him connect up to the grid again until a real electrician had straightened it all out.

It's true that this stuff is not that hard to learn if you study it. But an architect friend likes to say that the building code is written in blood, meaning that much of it is there because confident dumbasses managed to kill enough people that they had to add a new rule. If people are prepared to learn the rules and appreciate why they're there, I'm all for it. But if they do it coming from a place of proving that they're not "so afraid of residential power", that's a terrible way to approach it.

[matt-p]

To be fair I'd be quite a bit more relaxed working on 120v. Very supprised these machines don't run on dc internally?

[wpietri]

In the older ones, it's almost all AC. One giant transformer, a couple of different voltages. Possibly with "high tap", a way to compensate for wall current with lower than expected voltages. The past is another country.

[pupdogg]

You can run a setup of 8x 4090 GPUs using 4x 1200W 240V power supplies (preferably HP HSTNS-PD30 Platinum Series), with a collective use of just around 20-amps, meaning it can easily run on a single 240V 20-amp breaker. This should be easily doable in a home where you typically have a 100 to 200A main power panel. Running 4x 1200W power supplies 24 hours a day will consume 115.2 kWh per day. At an electricity rate of $0.12 per kWh, this will cost approximately $13.82 per day or around $414.72 per month.

FYI, I can handle electrical system design and sheet metal enclosure design/fabrication for these rigs, but my software knowledge is limited when it comes to ML. If anyone's interested, I'd love to collaborate on a joint venture to produce these rigs commercially.

[google234123]

In Cali isn't now like 0.5 per kWh :P

[pupdogg]

Wow!

[XMasterrrr]

Yeah, you get it. I am using 3x 1600w 240V Platinum PSUs from Superflower (They're the manufacturer of EVGAs PSUs. Top notch stuff, did my homework) I however decided against server PSUs like the one you suggested because the extra setup overhead + noise.

As I mentioned in my reply to OP, very doable as long as you do your research. The only thing I did not do was not doing the installation itself because I was not comfortable with it, but I pretty much had everything named to the contractor, and even how I would have gone about the installation process was exactly how he did it.

Hit me up on Twitter or Email, we can chat ideas about this venture

[swader999]

You'll have helicopters over your house.

[pupdogg]

Believe it or not, a buddy of mine was dealing with crazy high power bills for three months and couldn’t figure out why. He tried everything to cut back, but nothing worked. Finally, he called an electrician, who found a hidden short in a 30-amp circuit that was constantly drawing power without tripping the breaker. After fixing the issue, his bills went back to normal—no helicopters were involved during the process!

[fennecbutt]

And all that heat was going...where?

[w-ll]

Good thing i can see them coming, and happy to offer anyone a cup of coffee/tea to understand im just the biggest nerd in the hoa.

[defrost]

Already have to deal with Robinson R22's during mustering season, a few more won't hurt.

[orbital-decay]

>Anything around ~2k watts on a single circuit breaker is likely to flip it

I'm curious, how do you use e.g. a washing machine or an electric kettle, if 2kW is enough to flip your breaker? You should simply know your wiring limits. Breaker/wiring at my home won't even notice this.

[trillic]

My kettle only pulls 1500W, as do most in the US. Our water just takes longer to boil than in Europe. My washer / dryer has its own 30a breaker as does my Oven as well as water heater. My garbage disposal has its own 15a breaker.

Boiling 1 liter takes like 2 mins. Most Americans don’t have kettles because they don’t drink tea.

[immibis]

Americans do not have electric kettles and need special circuits for electric clothes dryers.

[lolinder]

We have an electric kettle in the US and it runs just fine drawing 1500W.

You're correct that the dryer is on a larger circuit, though.

[beAbU]

> and it runs just fine drawing 1500W.

You think that this is "just fine" because you've never experienced the glory that is a 3kW kettle!

[rootusrootus]

I just get 99C water from a tap next to my kitchen sink. Why do people still use kettles?

[dymk]

Because they don’t have a spare few grand for an instant hot plus installation

[fragmede]

Japanese water boilers keep it boiling so there's no wait at all!

[blibble]

I get bored and tend to wander off waiting for it to boil at 3kW

1.5kW must be absolute agony

[lolinder]

I mean... yes, I don't sit around waiting for the kettle to boil. But if I fill it and start it first the water is already boiling by the time I get everything out, so it's not like any time is wasted as is.

[jamesbfb]

Huh, what?! Mega TIL moment for me as an Australian with an electric kettle and dryer plugged into whatever power socket I wish! Reminds me of this great Technology Connections video: https://youtu.be/jMmUoZh3Hq4?si=3vSMHmU2ClwNRtow

[Yodel0914]

I'm pretty sure in our current rental the kitchen and laundry are on the same circuit, which means I'll often have the dryer, washing machine, kettle, toaster and microwave drawing power at the same time. It's never been an issue.

[XMasterrrr]

What kind of a dryer? Because that cannot be right. Dryers require a 30amp 240volt dedicated breakers by code in most counties in most states nowadays.

[Yodel0914]

It's a bosch heat pump dryer, but previously we had a traditional vented dryer.

I've never seen a dedicated circuit for dryers in Australia, and I've lived in probably a dozen different properties. Ovens, aircon, hot water, bathroom heat lamps often have dedicated circuits, though.

[HeatrayEnjoyer]

Insane

[XMasterrrr]

If that is true and OP is not just confused, he should sue his landlord, and I am not even kidding!!!

[HeatrayEnjoyer]

Why would regular sockets need to supply that much juice

[mattnewton]

I rent an old Victorian. I have one breaker line for the fridge and microwave and one line for basically everything else.

If that wasn’t the limit though, the fact that the machine is currently a space heater at 2 liquid cooled 4090’s would be.

[nathanasmith]

I heat water on the stove top which is plugged into a 240 volt outlet.

[throwthrowuknow]

Not speaking from direct experience building a rig like this but the blog post mentions having 3 power supplies so the most direct solution would be to put each on their own dedicated circuit. As long as you have space in your electrical box this is straightforward to do though I would recommend having an electrician do the wiring if you aren’t experienced with that type of home electrical work.

[gizmo686]

Even without space in the existing box, installing a subpanel isn't that much more of a cost.

[XMasterrrr]

Actually, putting each PSU on its own circuit is crazy dangerous. In the scenario of your suggestion, if one goes out, you are in for a fire. Highly recommend against that.

[gizmo686]

This might be one of those reasons why I'm not an electritian, but is it dangerous.

Back when I worked at a high-availability data center, all of our servers had dual psus, plugged into seperate circuits.

The transformer in the PSUs should electrical isolate the mains voltage from the low voltage side, so you aren't going to cause a short across the two circuts.

The only risk I see is a cascade failure, where the increased load on the second circuit causes its breaker to trip.

[throwthrowuknow]

Do they not have fuses? Use 15A breakers?

[bluedino]

Take your typical 'GPU node', which would be a Dell/HP/SuperMicro with 4-8 NVIDIA H100's and a single top high level AMD/Intel CPU. You would need 2-4 240v outlets (30A).

In the real world you would plug them into a PDU such as: https://www.apc.com/us/en/product/AP9571A/rack-pdu-basic-1u-...

Each GPU will take around 700W and then you have the rest of the system to power, so depending on CPU/RAM/storage...

And then you need to cool it!

[XMasterrrr]

Yeah, you get it. I have two PDUs with NEMA L6-30P plugin, and two NEMA L6-30R outlet. Each outlet is connected to it's dedicated 30A 240v circuit (which gives you a total of 60A, or 6000w).

Cooling is its own story... Man, figuring out this process was hella of a journey

[fennecbutt]

Is suppose this is an American view. Most places with 240 you can run anything up to 3kW per socket most of the time. But you can also get a sparky and go for a cheap high current socket install on 240 or even pay a bit more to get 3 phase installed, if you have a valid enough use case.

Hell most kettles use 3kw. Tho for a big server I'd get it wired dedicated, same way power showers are done (7-12~ kW)

[abound]

Not OP, but my current home had a dedicated 50A/240V circuit because the previous owner did glass work and had a massive electric kiln. I can't imagine it was cheap to install, but I've used it for beefy, energy hungry servers in the past.

Which is all to say its possible in a residential setting, just probably expensive.

[woleium]

Yes, or something like a residential aircon heatpump will need a 40a circuit too. Car charging usually has a 30a. Electric oven is usually 40a. There’s lots of stuff that uses that sort of power residentially

[slavik81]

Not the OP, but I hired an electrician to put a 30A 240V circuit with a NEMA L6-30R recepticle next to my electrical panel. It was 600 CAD. You can probably get it done cheaper. He had to disconnect another circuit and make a trip to the hardwate store because I told him to bring the wrong breaker.

[XMasterrrr]

Yup, that's exactly what I had installed.

[GaggiX]

I use a hair dryer that is a little bit more than 2kw, but I guess because of the 120V it would be a problem in the US.

16 amps x 120v = 1920W, it would probably trip after several minutes.

16 amps x 230v = 3680W, it wouldn't trip.

[sva_]

When my gf first came to Europe she brought her hairdryer from the US and plugged it in using an adapter that just reroutes the wires. She was unaware of the voltage difference (or thought the adapter would adjust it.) That thing started spewing fire pretty much immediately and I luckily quickly realized what she was doing and was able to pull the plug (I hadn't noticed that she brought her own hair dryer.) Luckily she wasn't pointing it at herself ...

[sandos]

This is funny as a european, since we have many, many groups where we reguarly will run 2kW, and some, loads. Really no issue, but I guess lower voltage makes it a problem.

[sixothree]

Yup. We typically have 20 amp breakers in living portions of the house and it's common practice for most devices to top out at 1500 watts. But from your description, you would still need three lines and three breakers. So. I'm not understanding your point.

[XMasterrrr]

Most outlets rely on 120v 15amp breakers here in the US, only washer/dryer/kitchenware require the higher end 240v breakers

[teaearlgraycold]

I’ve ran 3x L40S on a 1650W PSU on a normal 120V 20A circuit.

[XMasterrrr]

You've got a much stronger heart than me man

[teaearlgraycold]

Sometimes you just need to YOLO it.

[littlestymaar]

Then juste add a 32A circuit breaker to your electrical installation, it's not a big deal really.

[XMasterrrr]

Oh yeah, my original setup was an RTX 4090 + an RTX 3090, and I swear one night I had the circuit breaker trip more than 15 times before I gave up. I have a UPS so I would run to the box before my system shuts down. Most houses are equipped with 15amp 120v breakers, these should never exceed 1500w, and their max is 1800w but then you're really risking it.

So, as mentioned on the article, I actually have installed (2) 30amp 240v breakers dedicated entirely for this setup (and the next one in case I decide to expand to 16x GPUs over 2 nodes lol). Each breaker is supposed to power up to 6000w at ease. I also installed a specific kind of power outlet that can handle that kind of current, and I have done some extreme research into PDUs. I plan on posting about all of that in this series (part 3 according to my current tentative plans) so stay tuned and maybe bookmark the website/add the RSS feed to your digest/or follow me on any of the socials if this is something that you wanna nail down without spending a month on research like me :'D

[mattnewton]

Thanks! Yeah the 4090’s are very thirsty if you let them be, I haven’t played enough with throttling their voltage and how that affects perf. Looking forward to your articles.