[threwrfaway]

For those who dont know why this is important:

The loads are slowing down the generators that are burning a well metered amount of fuel to stay at 60Hz. This is a delicate balance since the phase angle must also be spot on.

If a generator and the local line disagree on f, phase or V, you have a short circuit.

If you lose a large amount of load, your generator will spin up with the excess fuel until the control system re-establishes the right amount of fuel.

But now your generators are out of sync! No worry, for small disturbances the dissipative losses sync everything up like syncros on a manual transmission.

But the disturance cant be too big!

Rotating machines are big and heavy, so the first line of defense is their inertia. But this is a finite (and precious) resource.

Contrary to belief, renewables, or generally speaking DC, makes things this stability problem worse. They generate large amounts of power while providing no inertia.

You'd think it isn't a big deal since the DC-AC converter can just synthesize whatever is needed. Heck just keep it rigid at 60 Hz with no phase change.

Well the later doesn't work - the rest of the grid is no longer at that phase and frequency so you got yourself a short.

Furthermore, the DC-AC converter, despite their manufacturers' promise, has no good way to establish what f and phase it should be at during a disturbance (and these magic codes are closed source, believe it or not)

Anywho, a large enough loss of load causes the grid to enters into unstable oscillations, causing protective relays to trip causing a zipper effect where the grid goes down.

Now restart will take a few days depending on the energy mix (fastest for hydro heavy)

Long story short - this is not a trivial problem, and the data-centers can't be allowed to just dump load willy nilly.

EDIT: made it clear that the grid killing disturbance is not caused by renewables; not exclusively anyway. Everyone has to play nice or the grid goes down.

[jillesvangurp]

Grid forming batteries and inverters are a thing. They can control the frequency on the grid. Just a matter of getting the right equipment.

The nice thing with data centers is that they are somewhat flexible. It's not a constant load. Data center operators can choose to reduce load. And if properly engineered, they could do so automatically based on signals from the grid.

The issue with outdated grids is that it relies on technology (spinning mass) that's at this point a century old. Which makes it brittle against outages like you describe. The solution is not more spinning mass but batteries and renewables to take the place of that spinning mass. A battery can respond to oscillations in milliseconds. If you then add flexible load that can spin up/down based on the amount of available power, you gain a lot of stability.

[alexey-salmin]

> The issue with outdated grids is that it relies on technology (spinning mass) that's at this point a century old. Which makes it brittle against outages like you describe. The solution is not more spinning mass but batteries and renewables to take the place of that spinning mass. A battery can respond to oscillations in milliseconds. If you then add flexible load that can spin up/down based on the amount of available power, you gain a lot of stability.

Why the spinning mass technology being "century old" (more like "millennia old" but anyways) is a problem somehow? The Newton's First Law didn't change much in the time that has passed.

Anyone who tried to "respond to oscillations in milliseconds" knows how hard that problem is because the force you apply is integrated twice before it takes effect. Try stabilizing a swing by pushing it's forward when it's behind the equilibrium point and pushing it backward when it's ahead of it. Now imagine a grid of swings connected by rubber bands and a distributed system of independent actors responding to oscillations. There are much more ways in which this system can diverge rather then converge.

Time may prove me wrong, but the arguments like "spinning mass is old therefore should be replaced" certainly won't.

[jillesvangurp]

> Newton's First Law didn't change much in the time that has passed.

No, but technology has moved on quite a bit. Heavy fly wheels are no longer the state of the art here.

A few tens of GW of battery capacity (i.e. a few dozen nuclear plants worth of capacity) that can switch on/off in milliseconds can do a lot for grid stability. That's part of the reason why grid operators are rolling out so much batteries. It's not necessarily about supplying energy for a very long time but about smoothing out peaks and dips in energy supply and demand and responding more or less in real time to that.

This stuff is basically being rolled out at industrial scale in a lot of places. Australia, China, etc. pretty much run increasingly on mostly renewables. This is no longer as speculative as it would have been ten years ago.

Yes, there are engineering challenges with rolling that stuff out in a lot of places. And even more policy and regulation challenges. Actually that is, by far, the #1 challenge in places like the US and Europe. Grid operators are simply structured and incentivized wrong to deal with this stuff efficiently. Texas is actually not doing too bad relative to e.g. California. But they clearly have some challenges still.

[cogman10]

> A few tens of GW of battery capacity (i.e. a few dozen nuclear plants worth of capacity) that can switch on/off in milliseconds can do a lot for grid stability.

I think this is a misunderstanding of the problem.

Now, don't get me wrong, I believe flywheels are no longer involved but at one point they were for batteries and solar. Not to store energy, but rather to form the output voltage and to give the correct "inertia" of the waveform to maintain a correct phase with the grid. Prior, both batteries and solar were frequency followers. They'd look at the previous peaks and valleys to determine what their output voltage (or resistance) should be. If the input voltage fell too far, both solar and batteries would cut off to avoid damaging equipment on the grid. This is part of why the winter storm killed the texas grid (to my understanding) the voltage dipped too low which ultimately caused renewables to shut off completely to avoid damaging the grid. That all was somewhat of a cascading disaster.

Flywheels have been used as an inertia source to allow for both solar and batteries to act more like a hydro or fossil fuel generator. That's the grid forming technology. I believe (can someone verify?) that there are now all digital versions of this. But it's delicate software. Getting it wrong can do really bad things like destroying other generators or breaking expensive fuses.

[SR2Z]

I would imagine that any single renewable plant could handle that - the the frequency drifts out of sync, it's only the inverter that will blow up.

It's an expensive part to be sure, but not so expensive that a plant couldn't keep a few spares.

[TheJoeMan]

Respectfully, I believe you have confused "new" with "state of the art". China is likely using batteries because their battery production is subsidized, Australia's trading partner is China so basically equivalent.

Perhaps a bad analogy, but it seems these battery systems attempt to stabilize by "pushing", whereas spinning mass can also work through "dragging" the phase. So eventually, you will have just a few rotating masses setting the freq and phase, with more and more systems tracking that and pushing, which seems like a recipe for chaotic equilibrium.

[jillesvangurp]

No, they are doing that because they have a ten year head start on the rest of the world. The US & EU not having done their homework is entirely on them. The US is choosing to pump a lot of funding into fossil fuels instead. You could say that China is simply spending smarter.

The rest is just economics. We might suck at making batteries. But we suck even more at making new gas/coal plants or fueling those cost effectively.

Anyway, this stuff is being deployed by the hundreds of gwh per year now. Much of it in China but some non trivial amounts in places like Texas and California as well. As a result, the grid is actually getting more stable, not less stable.

[nutjob2]

> Respectfully, I believe you have confused "new" with "state of the art". China is likely using batteries because their battery production is subsidized, Australia's trading partner is China so basically equivalent.

This is comically wrong. You're just making things up. Australia is at the forefront of this new tech due to necessity and some good luck. Scroll to the bottom of this article and have a look at the graph. It shows Australia has 5 times the grid forming battery infrastructure deployed or under construction than China or the US, or pretty much any other country. It is very much SOTA, needed for a grid that is a rapidly changing mix of rooftop solar, hydro, coal, gas and wind spread over a country the size of the continental US. Rotating masses are not going to cut it.

https://arena.gov.au/blog/australias-grid-forming-battery-re...

[joquarky]

> nuclear plants worth of capacity) that can switch on/off in milliseconds

Nuclear has this little problem getting in the way of switching on/off quickly:

https://en.wikipedia.org/wiki/Xenon-135

[expedition32]

Yes we know the solution is infrastructure. The question is who gets to pay for it.

In America that seems to be the dying small town whose only economic value is cheap land.

[jillesvangurp]

Infrastructure is an investment with an ROI. Fuel is an expense that delivers energy once and that is never recovered. The only relevant debate here is on how to finance the relative attractive investment for renewables and over how long it delivers that ROI.

That varies per region but this is a sector that is now about 2x the level of investment for fossil fuel related infrastructure. A lot of which is increasingly looking like the ROI might never happen.

[joquarky]

> over how long it delivers that ROI

In the US, this is moot if that time period is more than 3 months.

[namibj]

10+ MW voltage-source converters that can't do up to around 80% of their nominal capacity as mostly-reactive apparent power with stabilizing synthetic inertia scaled as desired/specified are a mostly software issue, stemming from lack of regulatory pressure incentivizing the engineering complexity of that.

Though if you want to do a smoothing action on real power flux you'll have to colocate battery capacity with the converter. Which to be clear is fairly cheap to do as long as you get compensated for the substantial frequency stabilization capacity this represents. I'm talking like 15~120 minutes at converter nominal AC power of battery capacity.

The first 10~20% of reactive power are almost free from the converter electronics, btw....

[nutjob2]

The software is being written along with the regulations and compensations and it's being rolled out in the real world, just as you described.

[peterus]

To add on to this overview, there are a new class of inverters in research called grid forming inverters which don't just follow the rest of the grid. One interesting technique is they can simulate the inertia of a traditional rotating machine/generator.

https://iten.ieee-ies.org/journal-featured-article/2025/grid...

[Retric]

That’s a simplified and somewhat outdated version, there’s a huge range of technology to mitigate each of these issues but infrastructure generally isn’t free. A turbine does provides inertia for free where you need to pay for a flywheel or battery system.

In the end a renewable heavy grid can be extremely resilient due to all the batteries, but smart battery systems need to be incentivize or mandated because ‘dumb’ batteries are cheaper.

[gblargg]

The ELI5 version is you're arm-wrestling someone and they suddenly let their arm go limp, so your arm slams down on the table since you can't react that fast.

[flopbob]

To build on this analogy, tug of war fits a bit better. Nothing dramatic happens if one person let go, but if half of one team just let loose at the same time without communication, bad things happen.

[cogman10]

I think an even better example is what happened to the Tacoma Bridge. You have an oscillating frequency and phase that needs to be precisely tuned in order for everything to work well. If something is out of sync wrong, you can end up with higher peaks, lower valleys, or flatter locations which can ultimately cause catastrophic failure.

In the Tacoma example, the input frequency continued to add onto the bridges motion which ultimately caused it to destroy itself. In an electric grid, a misaligned phase can cause excessive spikes (imagine 480V when you expect 240V) or the generators to ultimately burn themselves out because 2 generators are fighting with each other, one trying to raise the voltage while another is trying to decrease the voltage. The really tricky thing is that load (particularly inductive or capacitive loads) look almost exactly like an unaligned generator.

https://www.youtube.com/watch?v=XggxeuFDaDU

[gblargg]

Reminded me of this fun video of a guy spinning up a water turbine generator and getting it synchronized with to the grid the old-style needle phase meter, then connecting it successfully:

https://www.youtube.com/watch?v=xGQxSJmadm0

[altairprime]

I interpreted from that EU grid post mortem that individual generators are coordinated using out of band comms channels that aren’t the power grid itself. Am I mistaken that they do this? Is it done there, but absent here?

[WarmWash]

Keeping the build trend going:

Communication that detects the release and travels as close as possible to the speed of the natural signal in the rope, and is robust enough to recover without losing stability if the other team grabs the rope again while your "let go" signal is mid-flight.

One way to dampen this is to put a really strong guy on each side, with instructions to never let go of the rope. These are the flywheels of the grid.

[ksec]

Thanks. This is a great analogy.

[bryanlarsen]

That's not true. Modern batteries and inverters can be really good at providing synthetic inertia.

Proof by example: https://spectrum.ieee.org/grid-scale-battery-scotland

As the article says, this is difficult and adds significantly to the cost. But it is very possible.

(It also adds significantly to the cost of spinning mass generators too. Delivering 250% of nominal current in a short circuit situation requires beefing up a lot of components).

There are many others.

[namibj]

Actually large data centers at least if done in a vaguely alirack style architecture, can do this with a decent fraction of their nominal power for very little hardware cost, as reactive power and real power add up via Pythagoras (`apparent=sqrt(real^2 + reactive^2)`) to the apparent power (rms voltage times rms current, which is what the 60Hz electronics and 60Hz transformers care about). The first 10-ish % are nearly free.

And alirack style datacenters have large 3-phase converters between the grid and some 240 (nowadays often 350) V DC bus, with the battery banks directly (with just fuses and sometimes a little bit of balancing/nudging power (think 10% of battery power rating)) on the bus, and then the servers also directly consuming from that bus.

The large converters on the battery bus thus allow synthetically smoothing load transients to the grid using the batteries to smooth that power draw. This has just minor additional wear on the batteries and a small power efficiency impact from hitting through the batteries, both of which are easily paid by anything market-rate of providing that grid service. Because they already need the power electronics and batteries anyways, unlike a utility battery farm that at best can argue day/night load shifting of solar production as the reason for the electronics and batteries to exist.

In that same spirit it's also effective to put batteries on the DC bus (between MPPT and inverters) of large solar farms, because they need the electronics anyways and it's actually reducing the required inverter&transformer capacity of the solar farm by peak-shaving.

[TOMDM]

Would demanding that large spikey users of energy like data centers implement some sort of demand ramping/isolation from the grid in the form of a massive capacitor bank or flywheel generator between them and the grid help reduce the risk here?

[manarth]

The data centres aren't inherently spikey, in general use their consumption is reasonably predictable.

However, if a DC detects that the _grid_ is wobbly (voltage or frequency deviations) the DC will disconnect without warning, and switch to its batteries and generators.

The grid complains because it's suddenly lost hundreds of MW of load. For the DC to have isolation capability, it would need a load-sink which can consume roughly the same power as the DC in normal operation, and can take in that load at a moment's notice.

It's a hard problem to solve, and probably better managed at grid-level than DC-level.

[thelastgallon]

> The grid complains because it's suddenly lost hundreds of MW of load. For the DC to have isolation capability, it would need a load-sink which can consume roughly the same power as the DC in normal operation, and can take in that load at a moment's notice.

Thats why always have all the EVs be connected all the time (except the 20 mins they drive). EVs can provide demand as a service and take excess power whenever its available (instead of solar curtailment), and also provide an immediate source of load when events like this happen. Its a shame US is anti EV, it has the best systems at scale that can be leveraged to transform the entire energy ecosystem.

[TOMDM]

My intuition is that there would be a fairly stable base load, but doing something like switching on a new training run of a frontier model would be incredibly spiky, thousands of GPUs going from somewhat idle to 100% in seconds.

[WarmWash]

This is where a smart grid would really shine, because the utility could drop the electricity price to 0 or even negative and turn on 100,000 air conditioners/heaters to effectively give people free climate control to absorb the load imbalance.

[trogdor]

Minor note: when discussing electricity, abbreviating datacenter as “DC” is confusing

[manarth]

Haha good spot. And when the data centre is running on batteries, the DC uses DC. And if it's in the USA capital, it could be a DC using DC in DC.

[cucumber3732842]

>it would need a load-sink which can consume roughly the same power as the DC in normal operation, and can take in that load at a moment's notice.

Liquid rheostat. A big one.

[jampekka]

> Contrary to belief, renewables, or generally speaking DC, makes things this stability problem worse.

Is there such belief? My feel is that anybody to whom electrical grid stability has even crossed their minds know this.

[somewhereoutth]

So apparently grid forming or synthetic inertia can be provided electronically nowadays, but presumably it would not be too terrible to put a rotating mass in between the (e.g.) solar power source and the rest of the grid? So the electricity from the solar panels runs a motor that turns a generator. I suppose power is lost through friction and other inefficiencies, and you'd need to co-locate the solar panels into a big enough farm.

[wyldfire]

Some of us in Texas are all too familiar with the problem of balancing load with generation, the risk of a cascade failure causing a slow restart.

During winter storm Uri, they did a duty cycle where we only had power available for ~6-12 hours at a time on the days it was available. This was apparently to avoid that very problem.

So far as I know, the obvious mitigations like winterizing NG generation and/or peering with neighbor grids have not been performed.

[tenthirtyam]

I've often thought that the target* grid phase should be encoded into a high frequency signal, say at 50kHz. Generators without inertia can immediately switch to the required phase and inertial systems can work towards it or disconnect their outputs if they stray too far.

The problem in my mind is that, as it stands, the signal that everyone has to latch on to is itself affected by load and by different generators latching on with different time constants in a complicated feedback loop. Would having a single authoritative source be an improvement? Would that be a way to eliminate the need for inertia?

*I suppose the "target" phase would probably be based primarily on the output from the biggest inertial systems and take into account their ability to adjust phase and frequency.

[otterley]

Practical Engineering made an outstanding video on this: "What Really Happened During the Texas Power Grid Outage?" https://www.youtube.com/watch?v=08mwXICY4JM

[nutjob2]

Grid scale battery systems are also used to maintain proper synchronization.

[hippich]

When you say "short circuit" - is this when wires get hit and make and burn, or this is reference to some other stuff?

[analog31]

Short circuit is when two sources try to drive the same node in a circuit at different voltages or frequencies. In an idealized circuit (such as where the wires are assumed to have zero resistance), current flow from one source to the other is infinite. In a real circuit, current flow rises to a level where "something has to give." This could include heating the wires between the sources or tripping some kind of protection mechanism.

I'm not great with analogies, but imagine a train with two locomotives, and each one is set to run at a different speed. There will probably be a lot of screeching of wheels etc. To make multiple locomotives work on a single train requires engineering them to be synchronized with one another.

[mitthrowaway2]

Synthetic inertia has come a long way. An inverter knows just as well as a rotating generator what frequency and phase it should be generating, because it can mimic exactly the ideal equations of motion of a rotating generator with programmable inertia.

[digitalsushi]

Could the GPS provide a correct carrier wave for every electrical system to synchronize with?

[chaz6]

Is it feasible to tie the grid frequency to an external clock reference, such as TAI?