[1053r]

For the purposes of information security, the nameplate capacity is the correct number to consider for a very simple reason: we must defend as if hackers will pick the absolute worst moment to attack the grid. That is the moment when the sun is shining and it's absolutely cloudless across Netherlands, California, Germany, or wherever their target grid is.

At that moment, the attacker will not only blast the grid with the full output of the solar panels, but they will also put any attached batteries into full discharge mode as well, bypassing any safeties built into the firmware with new firmware. We must consider the worst case, which is that the attacker is trying to not only physically break the inverters, but the batteries, solar panels, blow fuses, and burn out substations. (Consider that if the inverters burn out and start fires, that's a feature for the attacker rather than a bug!)

So yes, not only is it 25 medium sized nuclear power plants, it's probably much higher than that! And worse, that number is growing exponentially with each year of the renewable transition.

This was probably the scariest security expose in a long time. It's much much worse than some zero-day for iphones.

A bad iPhone bug might kill a few people who can't call emergency services, and cause a couple billion of diffuse economic damage across the world. This set of bugs might kill tens of thousands by blowing up substations and causing outages at thousands to millions of homes, businesses, and factories during a heat wave. And the economic damage will not only be much higher, it will be concentrated.

[bramblerose]

The failure mode is much simpler: you don't need to physically break anything, you just need to drop 10GW of production from the grid (send a "turn off" command to all solar inverters) leading to a cascade of failures. Getting the grid back online is a laboreous manual process which will take (a lot of) time. Think https://en.wikipedia.org/wiki/Northeast_blackout_of_2003 or https://en.wikipedia.org/wiki/2021_Texas_power_crisis .

[tivert]

> Getting the grid back online is a laboreous manual process which will take (a lot of) time. Think...

It would be even more laborious and take more time to bring things back online if the attacker manages to damage or destroy equipment with an overload like the GP describes.

[msandford]

The "turning the grid up to 11" attack isn't really possible. I know it seems like it is, but the inverters will only advance frequency so much before they back off, the inverters will only increase voltage so much. Etc. Sounds scary, isn't practical.

Turning everything off when the panels are at peak output? That lets frequency sag enough that plants start tripping offline to protect themselves and the grid and it'll cascade across the continent in just a few minutes. Then you have a black start which might take months.

There's an excellent video on how catastrophic a black start is. https://youtu.be/uOSnQM1Zu4w?si=x0dA7X7-19CJm6Kf

[kortilla]

Months isn’t correct. Unless there was damage it could be recovered within a day.

[msandford]

Would love to know more about this. How would that happen? What's the process to bring it back up so fast?

The video has a lot of good info and seems compelling. During the Texas freeze many power company officials said the exact same thing, if the Texas grid went down it would have taken weeks to bring everything back online.

[cesarb]

> What's the process to bring it back up so fast?

It's called black start (https://en.wikipedia.org/wiki/Black_start), and power companies plan for it, and the necessary components are regularly tested. It's not a fast process, it can take many hours to bring most of the grid back up. We had last year a large-scale blackout here in Brazil, and a area larger than Texas lost power; most of it was back in less than a day.

> if the Texas grid went down it would have taken weeks to bring everything back online.

The trick word here is "everything". Every time there's a large-scale blackout, there's some small parts of the grid which fail to come back and need repairs. What actually matters is how long it takes for most of the grid to come back online.

[tlb]

Inverters may be protected against changing settings, but if you can replace the firmware it can likely cause permanent hardware damage. Which the manufacturer, perhaps under pressure from its government, can do.

[huijzer]

That doesn’t necessarily lead to a failure. In Texas a nuclear plant went down. About 1.2 (or 4 times that) GW (https://www.keranews.org/news/2023-06-22/with-temperatures-s...).

The grid stayed online. Likely thanks to grid batteries, see aforementioned link.

[deanishe]

Four units with a combined capacity of 1.2GW.

That's a far cry from 10GW.

[t0mas88]

The risk is not turning all solar installations "on maximum". That happens nearly every summer day between 1 and 2pm. Automatic shutoff when the grid voltage is rising can be disabled, but more than 9 out of 10 consumer solar installations in the Netherlands deliver their maximum output on such a day for most of the summer, not running into the maximum voltage protections.

The big risk is turning them all off at the same time, while under maximum load. That will cause a brown-out that no other power generator can pick up that quickly. If the grid frequency drops far enough big parts of the grid will disconnect and cause blackouts to industry or whole areas.

It will take a lot of time to recover from that situation. Especially if it's done to the neighbouring grids as well so they can't step in to pick up some of the load.

[more_corn]

Not if we have grid scale batteries. Solar shuts off, oh no. Sometime in the next four hours we need to get that fixed or something else up. Also flattens out the demand curve and allows arbitrage between the peak and valley.

[ale42]

Problem is, those batteries are not there (yet)...

[huijzer]

Don’t underestimate exponentials. Tesla produced 6.5 GWh of battery storage in 2022, 14.7 GWh in 2023, and will probably double again in 2024.

And other battery manufacturers such as BYD grow fast too.

[automatic6131]

Always underestimate exponentials: none exist in nature, they're just an early phase of an S curve (sigmoid, if you want the $10 word)

[huijzer]

What makes you think we're in the later stages of the battery S-curve currently? More generally technology-wise, what makes you think energy technology in general is an S-curve? Many situations are stacked S-curves. Global energy consumption, for example, looks like an exponential (no flattening yet) [1] if you plot it starting 1800.

Also, the start of an S-curve can be described by an exponential function, right? So it is an exponential.

My point more generally was to not underestimate processes which increase exponentially. Even if they flatten at some point, they can drastically change the world and fast. For example, iPhones and computer chips took off slowly, but once they started moving they took over the world. (Or do you not have multiple smartphones and computer chips in your house right now?)

And yes your point that it's all an s-curve is theoretically correct. But I think it's a semantic discussion. Next time I'll say "never underestimate the first half of an S-curve."

[1]: https://ourworldindata.org/energy-production-consumption

[ale42]

Which is kind of normal, we don't need infinite batteries ;-)

[mschuster91]

> We must consider the worst case, which is that the attacker is trying to not only physically break the inverters, but the batteries, solar panels, blow fuses, and burn out substations.

Power transformers have a loooooooot of thermal wiggle room before they fail in such a way and usually have non-computerized triggers for associated breakers, and (at least if done to code, which is not a given I'll admit) so do inverters and every other part. If you try to burn them out, the fuses will fail physically before they'll be a fire hazard.

[1053r]

This is true, especially for low frequency (high mass) inverters. The inverters that are covered here are overwhelmingly high frequency (low mass) inverters. We hope that they practiced great electrical engineering and layered multiple layers of physical safeguards on top of the software based controls built into the firmware.

Of course a company that skimped to the point of total neglect on software security would never skimp anywhere else, right? Right?

:crossed-fingers: <- This is what we are relying on here.

And even if they did all the right things with their physical safety, the attackers can still brick the inverters with bad firmware and make them require a high skill firmware restore at a minimum and turn them into e-waste and require an re-install from a licensed electrician at a maximum.

[mschuster91]

> Of course a company that skimped to the point of total neglect on software security would never skimp anywhere else, right? Right?

At least in Europe, product safety organizations and regulatory agencies have taken up work to identify issues with stuff violating electrical codes (e.g. [1] [2]) and getting it recalled/pulled off the market.

Sadly there is no equivalent on the software side - it's easy enough to verify if a product meets electrical codes, but almost impossible to check firmware even if you have the full source code.

[1] https://www.bundesnetzagentur.de/SharedDocs/Pressemitteilung...

[2] https://www.t-online.de/heim-garten/aktuelles/id_100212010/s...

[blkhawk]

> high skill firmware restore at a minimum and turn them into e-waste and require an re-install from a licensed electrician at a maximum.

Well not even high skill - for "security" reasons and to prevent support issues as well as to skimp on testing needed informations are often only accessible to a chosen few.

Paradoxically the effect of thes "security" concerns often mean that there are plenty of easily exploited methods in devices like that. And the only people that have them are the ones that you need to worry about instead of some 16 year old teenager finding it and playing blinkenlights with his friends parents house causing trouble for him but getting the hard coded backdoor taken out after the media got wind of it.

If I was dictator of infrastructure I would ban any non-local two way communication and would mandate all small grid storage solutions run off a curve flattening model thats uniform and predictable. Basically they would store first and only be allowed to emit a fraction of their storage capacity to the grid afterwards. Maybe regulated by time of day.

[idiotsecant]

This is wildly overstating the issue. Hackers are not going to break into hundreds of separate sites, compromise inverters, compromise relay protection, compromise SCADA systems, and execute a perfectly timed attack. Even if they did, these are distributed resources, they don't all go through a single substation and I doubt any one site could cause any major harm to any one substation.

Instead, they're going to get a few guys with guns and shoot some step of transformers and drive away.

The problem with infosec people is they tend to wildly overestimate cyber attack potential and wildly underestimate the equivalent of the 5 dollar wrench attack.

[g_p]

They don't need to break into separate sites though - the issue at hand is that a single failure in the centralised "control plane" from the vendor (i.e. the API server that talks to consumers' apps) can be incredibly vulnerable.

Here's a recent example where a 512-bit RSA signing key was being used to sign JWTs, allowing a "master" JWT to be signed and minted, giving control of every system on that vendor's control system.

https://rya.nc/vpp-hack.html

[Gud]

Most(more or less all of them) grid operators can operate their network remotely from a single control room.

I suspect most grids are extremely easy to hack(never tried, don't bite the hand that feed you etc).

Info sec is just a hobby of mine. I install high voltage switch gear for a living.

[applied_heat]

A lot of utilities have their own fibre since they own poles/towers and need it for tele protection anyway so they can have secure a real private network between control room and significant power plants

[TwiztidK]

> I suspect most grids are extremely easy to hack

I’d expect the opposite. All companies controlling equipment that is part of the “Bulk Electric System” have to be NERC CIP compliant and are audited regularly with large fines for non-compliance. Doesn't guarantee perfect (or even good security) but it’s more likely to be a priority.

[ufocia]

How do fines make things better? They confiscate resources that could be used to improve.

[aerostable_slug]

It also perverts incentives such that no utility will communicate perhaps helpful information to other utilities or government when said information can leave them liable for fines.

Until there's some kind of hold-harmless agreement, the various industry & government security information sharing groups can only be of limited effectiveness.

[applied_heat]

The management at the utility doesn’t want to be recognized for being a deficient operator that doesn’t meet standards, so they hire employees to ensure they are compliant

A fine is a black eye for a utility where people pride themselves on the reliability of the service they provide

[lll-o-lll]

Hurray! I have experience that may shed some insights. I worked on SCADA software (3 different ones), for about 15 years, started off as a Systems Engineer for an Industrial Power Metering company (but writing software), built drivers for various circuit breakers and other power protection devices, and wrote drivers and other software for IEC61850 (substation modelling and connectivity standard). I’ve been the technical director of one of these SCADA systems, and in charge of bringing the security to “zero trust”. I’ve been on the phone with the FBI (despite not being an American or in America), and these days I design and lead the security development at a large software company.

I’ve been out of the Power Industry/SCADA game for about 6 years now, and never had huge involvement with solar farms, so please take this with a large grain of salt, but here is my take. 15 years ago, all anyone would say about industrial networks was “air gap!”. Security within SCADA products was designed solely to prevent bad operators from doing bad things. Security on devices was essentially non-existent, and firmware could often be updated via the same connectivity that the SCADA system had to the devices (although SCADA rarely supported this; it was still possible). In addition, SCADA systems completely trusted communication coming back from the devices themselves, making it relatively simple for a rogue device to exploit a buffer overrun in the SCADA. After Stuxnet + a significant push from the US government, SCADA systems moved from “defensive boundary, trust internally” to “zero trust”. However, devices have a long, long service life. Typically they would be deployed and left alone for 10+ years, and generally had little to no security. Security researches left this space alone, because the cost of entry was too high, but anytime they did investigate a device, there were always trivial exploits.

Although SCADA (and other industrial control software), will be run on an isolated network, it will still be bridged in multiple places. This is in order to get data out of the system, but also to get data into the system (via devices, and off-site optimisation software). The other trend that happened over time was to centralize operations in order to have fewer operators controlling multiple sites. That means that compromising one network gives you access to a lot of real world hardware.

Engineers never trusted SCADA (wisely), and all of these systems would be well built with multiple fail-safes and redundancies and so on. However, if I were to be a state-actor, I’d target the SCADA. If you compromise that system, you have direct access to all devices and can potentially modify the firmware to do whatever you want. If there is security, the SCADA will be authorized.

I don’t think the security risks are overblown (they are overblown in what they think the real problems are). I think that as the systems have gotten ever more complex; we have such complicated interdependencies that it is impossible to deterministically analyse them completely. The “Northeast blackout of 2003” (where a SCADA bug lead to a cascade failure), was used as a case-study in this industry for many years, but if anything, I think the potential for intentional destruction is much higher.

[applied_heat]

I’m in this space, but plc io networks from Schneider and Rockwell are still “trust internally”, and some HMI or scada has to have read/write to them. At least Rockwell you could specify what variables were externally writeable whereas Schneider was essentially DMA from the network.

[1053r]

This isn't hundreds of separate sites that have to be hacked individually. This is fewer than 10 clouds with no security to speak of and the ability to push evil firmware to millions of inverters worldwide, where in a few years at the current rate of manufacturing growth, it will be 10s, and then 100s of millions of inverters.

Yeah, the potato cannon filled with aluminum chaff or medium caliber semi-automatic rifle can take down a substation. But this is millions of homes and businesses, which can all have an evil firmware that triggers within seconds of each other. (There will inevitably be some internal clocks that are off by days/months/years, so it's not like it will happen without warning, but noticing the warning might be difficult.)

And the growth in sales is exponential!

[ethbr1]

> medium caliber semi-automatic rifle

Technically, anything that can put a hole in an oil-filled transformer. https://en.m.wikipedia.org/wiki/Transformer_types#Liquid-coo...

You don't need to break it... just crack the radiator enough for all the circulating fluid to drain, then it overheats.

[EdJiang]

Important to point out this isn't just theory, it's actually happened (in the SF Bay Area!) with a regular rifle.

https://en.wikipedia.org/wiki/Metcalf_sniper_attack

https://www.npr.org/sections/thetwo-way/2014/02/05/272015606...

[ethbr1]

Also in the north GA mountains in the 1970s.

[applied_heat]

Any transformer over about 5 MVA will probably be equipped with a low oil level switch that de-energizes it

[jpc0]

If all you wantes was to kill the power I don't see the difference...

Sure the repair is easier/quicker but the economic damage was already done...

[applied_heat]

The lead time on a transformer that size is probably a year, so a year long outage has a lot more damage than a 2 week outage

[hn_throwaway_99]

While I agree that the important metric to consider is peak output and not average output, I would still guess that in a country like the Netherlands that peak output is nowhere near nameplate capacity.

[Retric]

You can get close to peak output just about anywhere, assuming the panels are angled rather than laying flat. You just can’t get it for very long in most locations.

[mapt]

The new method this past year that appears to be highly beneficial is to use various compass orientations of _vertically_ mounted panels. The solar cells got so cheap that every penny we spend on mounting hardware and rigid paneling now stings, and posts driven vertically into the ground which string cables tight between them are cheaper than triangles, way easier to maintain (especially in places with winter), and trade a lower peak (or even a bimodal peak) for a much wider production curve.

[immibis]

The "bad iPhone bug" scenario happened a few weeks ago, in the form of Crowdstrike. You underestimated the damages.

[verisimi]

Tldr; We can't talk about proper numbers cos hackers.