[Vox_Leone]

The fewer resources we dedicate to grid resilience and modernization, the harder black starts become. And as grids get more complex and interdependent, recovering from total failure becomes exponentially harder.

A rare but sobering opportunity to reflect on something we usually take for granted: electricity.

We live in societies where everything depends on the grid — from logistics and healthcare to communications and financial systems. And yet, public awareness of the infrastructure behind it is shockingly low. We tend to notice the power grid only when it breaks.

We’ve neglected it for decades. In many regions, burying power lines is dismissed as “too expensive.” But compare that cost to the consequences of grid collapse in extreme weather, cyberattacks, or even solar storms — the stakes are existential. High-impact, low-frequency events are easy to ignore until they’re not.

[the_duke]

Just to highlight this: the last significant power outage in Western/middle Europe was 2003. [1]

That's 20 years without any significant problems in the grid, apart from small localized outages.

It's not hard to start taking things for granted if it works perfectly for 20 years.

Many people don't even have cash anymore, either in their wallet or at home. In case of a longer power outage a significant part of the population might not even be able to buy food for days.

[1] https://en.wikipedia.org/wiki/2003_Italy_blackout

[citrin_ru]

> Many people don't even have cash anymore, either in their wallet or at home.

Even if you have cash many shops would not sell anything in case of a mass outage because registers are just clients which depend on a cloud to register a transaction. Not reliable but cheap when it works.

[bill3478]

Many supermarket chains (in the West at least) have satellite links at their major locations because they can't afford to close a store just because the local ISP had an issue.

The real question is how long can some of the smaller banks' datacenters stay up.

[inkyoto]

Firstly and most importantly, a cash register needs a power outlet. It is highly contestable that every single Western supermarket out there has a diesel generator down in the back / storage room that will kick in in an instant if a power outage begins.

Lest also forget the Crowdstrike drama where many supermarkets simply went dark, in some instances for nearly 24 hours, despite working communication links. But I digress.

[SchemaLoad]

Crowdstrike was an interesting one. Just as it was going down I went out to the supermarket and found half of the self checkouts had bluescreened. Then a few hours later they were all back and functioning again. The supermarket had remote management at a level below the OS that could restore the whole countries self checkouts rapidly.

[crote]

I would not be surprised if they simply booted an image from the network. It would significantly simplify maintenance, as for any change you'd just need to update a single image and push it downstream to an in-store management server. The individual terminals essentially become disposable.

[bell-cot]

> It is highly contestable that every single Western supermarket out there has a diesel generator down in the back / storage room that will kick in in an instant if a power outage begins.

Literally true. However:

- If it takes them 10 minutes to fire up the generator, then 5 minutes to restart the network and registers, that is no big issue (in a many-hour outage)

- At least in my part of the USA, many supermarkets do have generators - because storm damage causes local outages relatively often, and they'd lose a lot of money if they couldn't keep their freezers and refrigerators powered. Since the power requirements of the lighting and registers are just (compared to the cooling equipment) a rounding error, those are also on generators.

[IIsi50MHz]

Plus, there are backup-power lories and refrigerated trailers. If your shop doesn't have enough backup power for duration, you might see several of these pull into the carpark all at once. If not all of the chillers can be powered, shop's staff will schlep stuff to the refrigerated lories.

Seen it done in USA, for a Target next to a Kroger grocer. Kroger lost everything that needed cold after reserve either ran out or wouldn't start, but Target had a contingencies contract and lost no product.

[prmoustache]

Well,in my experience it was the case for the 2 largest supermarket chains. We lost electricity at 12:30pm and only got it back during the night at 3am.

But both major supermarkets nearby worked on diesel generators and payment by card worked flawlessly. I guess they had satellite connection.

It might have been more complicated in small villages but people living in rural areas ually still use a lot of cash.

[swiftcoder]

In my local area of Spain/portugal, 2/3 supermarkets and 2/3 gas stations had generators up and running within a couple of hours. We’re pretty rural, though - I don’t know that urban areas faired as well

[diggan]

We're outside Mataro, had to make a trip into Barcelona yesterday. I'd say most gas stations north of Barcelona/Maresme area were 100% offline, some (we found only two, from 6 visited) gas stations still had operational pumps but huge queues and cash only. None of the TPVs seemed to work anywhere in the afternoon yesterday here, even the battery powered/mobile network ones.

[ricardobayes]

That's true, I went shopping cca. 4-5 hours after the blackout started and had no issues, even card transactions worked. Whoever designed the retailer, they clearly had this scenario in mind. Even the "self-service" computer kiosks all worked.

[brewdad]

In a multi-day event like we are talking about here, couldn't a shop owner revert to a paper ledger? I mean, it would suck and transactions would take much longer but if the alternative is people starving or having your inventory looted by a desperate mob, a nineteenth century solution seems preferable.

[ryoshu]

They can and do. They will also do deep cuts on prices/give away for free for refrigerated and frozen goods because those will just get tossed and neighborhood goodwill is still a thing in some places.

[chihuahua]

There was a 4-day power outage here (Seattle suburbs) last fall, and one of the auto parts store made an effort to serve customers even though they didn't have any power. I paid cash, and I forget whether or how they did credit card transactions (possibly by writing down CC numbers on paper.) They made a lot of phone calls to a different store to get prices for items.

[spiddy]

I'm located in Barcelona, and yesterday lot of transactions on mini markets / pharmacies were not possible because the item prices were unknown, adding to the fact there was no phone lines available to reach out.

[miki123211]

And in many places in Europe, registers are literally required by the government to be this way, for VAT fraud avoidance reasons.

[ricardobayes]

It's probably just Hungary though, I don't know of any other country mandating an always-online cash register.

[pta2002]

Portugal has mandatory electronic receipts. By this I don't mean email receipts, I mean that all receipts have a code on them that is then also available to be looked up on the government's side (e-Fatura is the app for this) for tax reasons. I think it's fairly simple though, just registers the total amount, the seller, and how much VAT was paid.

However I assume this can work offline with the data being uploaded later though, as basically all the small supermarkets and shops were still open here (_incredibly_ chaotic though), and on the big supermarkets card payments were working (TBF, even the free wifi was working there, I guess they probably have some satellite connection).

[the_duke]

It's an EU regulation.

[cogman10]

> Many people don't even have cash anymore, either in their wallet or at home. In case of a longer power outage a significant part of the population might not even be able to buy food for days.

So, what's really interesting is that these sorts of social collapses have happened. In fact, they often happen when natural disasters strike.

When they do happen, mutual aid networks just sort of naturally spring up and capitalism ends up taking a backseat. All the sudden worrying about the profits of Walmart are far less important than making sure those around you don't starve.

As it turns out, most people, even managers of stores, aren't so heartless as to let huge portions of the population starve. Everyone expects "mad max" but that scenario simply hasn't played out in any natural disaster. In fact, it mostly only ends up being like that when central authority arrives and starts to try and bring "order" back.

You can read about this behavior in "A Paradise Built in Hell" [1].

[1] https://en.wikipedia.org/wiki/A_Paradise_Built_in_Hell

[holoduke]

Well if the situation happens that people can't buy food, things will easily become nasty quickly. People will break open stores and use violence to get what they need. So cash money won't really help a lot in a serious situation.

[ricardobayes]

The "civism" was well noted in the Spanish case, as far as I know, the whole country passed this incident without a single case of security issue. During the whole thing, people were extremely kind and polite. People sitting in dark bars was kind of "funny" how people clinging to their normal lives even in emergencies.

[prmoustache]

Shop owners would just take notes on who buy what and let people go with the items.

Looting only ever happens when areas hve started being evacuated and most shop owners + law enforcement are elsewhere.

[71bw]

And then take it up the butt from the taxman once he finds out.

[awesome_dude]

But but but, muh crypto is the future </sarcasm>

I often wonder if we should leave energy/telecommunications in a state where they can and do fail with some degree of frequency that reminds us to have a back up plan that works.

I had thought that the (relatively) recent lockdowns had taught us how fragile our systems are, and that people need a local cache of shelf stable foods, currency, and community (who else discovered that they had neighbours during that time!)

For something like this, a local electricity generation system (solar panels, wind/water turbines, or even a ICE generator) would go a long way to ensuring people continued to have electricity for important things (freezers)

[rini17]

There are now ubiquitous wireless POS terminals for card payments that can be recharged from emergency sources of electricity(like cars). As long as the mobile internet works it's possible. Of course this only little alleviates the disruption.

[userbinator]

As long as the mobile internet works it's possible.

If we're talking about a situation where the grid goes down, the mobile internet is most definitely not working.

[jadyoyster]

Surprisingly mobile networks seemed to stay up in Portugal. I'm not sure to what extent and if they lasted for the whole duration of the blackout. They definitely limited consumer use though.

[20after4]

Most mobile base stations have a limited backup battery and some have generators on site. I'd expect telephone infrastructure to have 24-48 hours of backup in the USA and I don't know why Europe would be much different.

[jhugo]

Population density is pretty high in many European inner cities. Most of the cell sites around here are on top of apartment buildings and I doubt they have a genset. Here in central BCN the mobile network was completely offline within an hour or two of the power going out.

[JaumeGreen]

I love in Spain and in my zone I didn't have Internet access until after 22:00, and it was sloppy at best.

0:49 and light came back, and woke us for a moment.

So yeah, you need local first POS applications.

[prmoustache]

Surely there are still card POS systems that can buffer transactions? Sure you loose some part of the system like payment authorizations but the potential loss of money is lower than closing shop.

[genewitch]

> a significant part of the population might not even be able to buy food for days.

And who's fault is that? Why did europe allow this?

Why will the US allow this, eventually?

[petertodd]

> And who's fault is that? Why did europe allow this?

In Spain it's now illegal to pay with cash for transactions over 1000EUR. Absurd.

In Norway they recently made it mandatory in most circumstances to accept cash for transactions up to 20,000kroner (~1700EUR): https://www.norges-bank.no/en/topics/notes-and-coins/the-rig...

[ricardobayes]

The 1000EUR limit doesn't apply between private individuals; for businesses, you will find many other European countries also don't take large cash payments, for security or convenience reasons. E.g. you can't buy a car from a dealership for cash in "cash is king" Germany either. They expect a wire in almost all cases.

[sapiogram]

I don't think anyone is going to starve to death because they can only spend 1000EUR...

[ithkuil]

I don't know but I find it very practical to not carry piles of cash in my pocket and home and know that we're less likely to get robbed just because of the cash we have.

I don't know how true the relationship between the cashless lifestyle and safety actually is, but it works and I feel ok; I'm not sure that the prospect of a few hours of national blackout once in 20 years will make me change my mind significantly.

[j1elo]

Today I was able to walk into a grocery store, pay for food, and go home to have a warm lunch (having a gas stove also helped tremendously). The matter was having a 10€ note at home. Not what I'd call "piles of cash".

As an added benefit, no bank knows where I bought and when, which I find is a great advantage over the alternatives. (I also use Gpay; this comes from someone who just found a good middle point without forgetting about the more reliable, physical and privacy friendly option)

[ithkuil]

I think I have 10€ laying around at all times, possibly in loose change between my home and car. I do not always walk around with that in my pocket and I never have more than 500€ at home.

I didn't mean literally zero cash, but once the bulk of your transactions are by card, you don't need to constantly go to the ATM and replenish your cash reserves

[j1elo]

And I myself didn't mean that I happened to have a measly 10€ note at home, I normally have around 300 minimum, to spend organically on purchases.

Of course I get that carrying coins and notes is cumbersome, but if we've managed to live all through the 80's and 90's with it, I think we can manage to keep doing it. 100% digital money is giving up on a huge level of self-determination and privacy that I wouldn't feel comfortable with, but I guess as newer generations grow up already pre-adoctrinated and not being able to compare the before-and-after, in the end society will end giving up.

[layman51]

It does seem safer to not carry cash. However, I remember around May 2024 that there were some reported incidents in Chicago were in the early morning hours, there were some groups of robbers who would force victims to do something I found especially worrying: they would be forced into resetting their phone password and logging into their mobile banking app. I’m not sure what ended up happening in these cases.

[artursapek]

Cash is extremely easy to conceal. fear of burglary is a ridiculous reason not to have some saved for a rainy day

[AStonesThrow]

More or less.

I once used an aggregator app to summon a handyman to my place. My request was simple: move two pieces of furniture around my very small apartment.

So I find a reputable service within the app, I schedule it, and they send a guy. He shows up to my door breathless, with some kind of sob story about a vehicle breakdown. I dismiss that out of hand and he gets to work. He did a fine job and it didn't take very long.

Then we get to the point of settling up, so I announce I'm going to pay in the app. He looks really disappointed and says he usually takes cash. I realized at this point that he was ready to shake me down, and also he would incidentally be discovering where I stashed my cash, when I reached for it with him there in the room. So disappointed. So I send the money out in the app and I show the confirmation screen to the guy. And I felt so bad that I followed up with a tip in the same fashion.

But at the end of the day he was just a garden-variety cash-in-hand scammer and I had no reason to feel guilty, because I had unwittingly outwitted him by trusting the app. And the company had no qualms about it.

Another time, I had a very short cab ride to the laundry. And it did not take long for the driver to spin a gigantic tale about his auntie addicted to gambling used up all their savings and they was really hurting for money. I was shifting uncomfortably wondering why I was hearing this. So the cabbie parks the car and his POS machine shuts off. He's like "oh it's out of order" so here he is, shaking me down and expecting me to go fetch cash to put in his grubby hands.

I stared him up and down, started taking photos, and got out of there. I discussed with dispatch. They said if he's not accepting cards and I intended to pay by card, I owe him nothing.

So again a cash-in-hand sob-story scammer was foiled. The cab service was crazy enough to assign him to pick me up additional times. This is why I ride Waymo, folks!!!

[phil21]

This might be one of the most paranoid things I've ever read on HN.

The laborer was simply trying to actually get paid vs. deal with the overhead of the app. Somewhat shady perhaps - since it routes around the company taking their cut for finding him the work, and likely avoids taxes. I've paid these sorts of guys cash every single time I've used such a service and exactly zero of them have "shook me down" or cared where I stored the money. They make so little already I'm happy to help them out with a smile.

Cabbies simply want cash for pretty much the same reason. They get charged an astronomical "service fee" by the cab company, and likely are avoiding taxes as well. I agree that such a situation is more shady in general, but I've actually had (NYC) cops side with cabbies on this topic and force me to go get cash at the ATM or get arrested. I also use car services now over cabs whenever possible due to this reason - mostly for convenience, never out of fear of being robbed though.

The chances of you getting mugged/stolen from for using cash are just the same as the chances of you getting mugged for no apparent reason walking home. Perhaps the collective dis-use of cash has reduced these odds, but you specifically is utterly irrelevant.

[artursapek]

Lol what? How do you know either of these guys were trying to rob you? They prefer cash because they can take it under the table and not report it for taxes. Everybody knows that.

[olyjohn]

Why will the US allow this? Because there's too much money to be made by middle men who want their cut of the transactions.

[emporas]

We will use better technology for electronic transactions. Most of banks worldwide still use COBOL for most (all?) their software infrastructure.

You can do as many electronic transactions as you wish without internet or electricity, provided you have something with charged battery. Problem is the transaction cannot be verified without internet, but when internet gets restored, all transactions can be applied.

That technology exists for more than a decade, so banks will implement it in 20 or 50 years. Most sane people will not wait patiently for half a century till some software engineer implements electronic transactions with COBOL, and we will use some kind of blockchain much sooner than that.

[shallichange]

I see no problem with COBOL. I'd rather have my credit card transaction be processed with 40yo well tested COBOL than Java from the newest intern using copilot.

[sgt101]

> Most of banks worldwide still use COBOL for most (all?) their software infrastructure.

Nahhh - some banks have some parts of the infrastructure in COBOL. Specifically larger retail banks often have their ledgers in COBOL. Most of them want rid and are actively getting rid. Most places have had programs to root COBOL out since before 2000, but there are residual implementations remain. The ledgers are the hardest place to deal with because of the business case as well as the awkwardness. Basically there's not much of an advantage (or at least hasn't been) in modernizing so keeping the thing going has been the option. Now people want to have more flexible core systems so that they can offer more products, although not so sure that customers want this or can consume it. Still - it supports the idea of modernisation so not many people are keen to challenge.

The most common big implementations I come across are in Java.

[emporas]

They use Go as well, I know someone who writes it for banks, but most of their infrastructure is written in COBOL. There are some sources [1], and some people have told me the same in person, not in exact numbers or percentages, but roughly the same.

Anyway point remains, electronic transactions with no internet or electricity is a solved problem, and banks don't want to solve it or they can't due to incompetency or maliciousness.

Currency transactions worth their weight in gold, it is of utmost importance for transactions to always be published to a central authority right away. If they don't have to be published, they should not exist at all. Imagine people buying stuff without anyone knowing right away! That should never, ever exist, for any reason.

[1] https://thenextweb.com/news/ancient-programming-language-cob...

[smackeyacky]

It’s not incompetence it’s regulations and inertia. Getting your central banking system past the regulators in most countries is difficult enough that it shades the effort required to replace the core systems.

[20after4]

> Imagine people buying stuff without anyone knowing right away! That should never, ever exist, for any reason.

I can imagine it and it happens all the time. Your version sounds very dystopian.

[LunaSea]

Anti tax evasion and money laundering measures

[roughly]

Yeah, this is the turkey’s dilemma - life on a farm is a lot better than life in the wild for 51 out of the 52 weeks of the year.

Most of our modern economy and systems are built to reduce redundancy and buffers - ever since the era of “just in time” manufacturing, we’ve done our best to strip out any “fat” from our systems to reduce costs. Consequently, any time we face anything but the most idealized conditions, the whole system collapses.

The problem is that, culturally, we’re extremely short-termist- normally I’d take this occasion to dunk on MBAs, and they deserve it, but broadly as a people we’re bad at recognizing just how far down the road you need to kick a can so you’re not the one who has to deal with it next time and we’ve gotten pretty lazy about actually doing the work required to build something durable.

[Xelbair]

"Just in time" is a phrase I hate with vehement passion. You aren't optimizing the system, you're reducing safety marigns - and consequences are usually similar to Challanger.

This is a solution that teenager put in management position would think of(along with hire more people as solution to inefficient processes), not a paid professional.

Systems like electric grid, internal water management (anti-flood) shouldn't be lean, they should be antifragile.

What's even more annoying that we have solutions for a lot of those problems - in case of electric grids we have hydroelectric buffers, we have types of powerplants that are easier to shutdown and startup than coal, gas or wind/solar(which cannot be used for cold start at all).

The problem is that building any of this takes longer than one political term.

[exe34]

> You aren't optimizing the system

Of course not, they're optimising shareholder profit.

[Marsymars]

> Systems like electric grid, internal water management (anti-flood) shouldn't be lean, they should be antifragile.

How do you make those systems antifragile rather than simply highly resilient?

[baq]

Things which can’t self improve can’t be antifragile by definition. NNT alludes to this multiple times - systems together with processes and people running them can be antifragile, but just things cannot.

I postulate the grid as a whole is antifragile, but not enough for the renewable era. We still don’t know what was the root cause of the Spanish blackout almost 24h after it happened.

[shermantanktop]

Step 1: read a book by nnt

[Marsymars]

Yeah, I have, that’s why I’m asking the question.

[signatoremo]

JIT isn’t about reducing safety margin. It was pioneered by Japanese companies, namely Toyota. They are known for risk adverse, safety first.

> This is a solution that teenager put in management position would think of(along with hire more people as solution to inefficient processes), not a paid professional.

What kind of comment is this? Toyota has been using and refining it for decades. It wasn’t invented yesterday by some “teenagers”. Such a state of HN’s comment section.

JIT is definitely not perfect as exposed during the Covid period, but it isn’t without merits and its goal isn’t “reducing safety margin”.

Then we have JIT in computing, such as JVM.

[phil21]

> its goal isn’t “reducing safety margin”

Sure it is. That's exactly how it achieves the higher profitability. Safety margin costs money. Otherwise known as inefficiency.

Slack in the system is a good thing, not a bad thing. Operating at 95% capacity 24x7 is a horrible idea for society in general. It means you can't "burst mode" for a short period of time during a true emergency.

It's basically ignoring long tail risk to chase near-term profits. It's a whole lot of otherwise smart people optimizing for local maxima while ignoring the big picture. Certainly understandable given our economic and social systems, but still catastrophic in the end one day.

[Xelbair]

It literally is reducing safety margin(buffers) of a whole distribution system by definition, and it is also being applied in places where it does not fit - like systems that should be resilient to disruption and/or anti-fragile.

I would expect a paid professional in management discipline to be aware of such nuance but alas proven wrong again.

[LorenPechtel]

Challenger wasn't really about cutting safety margins, but about kicking the can on a known problem: blowby in the motor joints. It was a gut feeling by the engineers that the problem was related to temperature, but there was enough of a random element to it that there was nothing specific to point to.

That should have been enough to scrub anyway, but there was clearly political pressure to launch.

I do agree that they need to specifically design anti-fragile.

[maxerickson]

What are some examples of modern system collapse?

We've had substantial disruptions, but they've not been particularly irrecoverable or sustained.

[exe34]

For the people who died of normally preventable death during covid while the health services were overwhelmed, the damage is irrecoverable. The chips shortage lasted years. Every year we become more, not less, dependent on the supply chain working. Every year we become less, not more, resilient.

[maxerickson]

I don't think it's crass to separate the deaths that occurred from a novel disease from the impact it had in society. In the medium term, it's a blip, never mind the long term. There's a huge chunk of society that thinks there was a huge overreaction!

The chips shortage has been difficult, but it's also been little more than an inconvenience when you look at it in terms of goods being available to consumers or whatever.

[tehjoker]

That chunk is heavily influenced by the propaganda that over a million dead people isn't a big deal. The propaganda is economically incentivized because slowing down the economy is bad for business even if it protects human lives.

[maxerickson]

Do you consider this significant failure a collapse?

I fell into the other poster's trap, talking about something emotionally charged that isn't really responsive to what I said initially.

[LorenPechtel]

The post you are replying to is not talking about the Covid deaths, but rather about the deaths from other causes triggered by the Covid disruptions. When a trauma case dies of the lack of a ventilator because they're all in use on Covid patients. When the trauma case bleeds out at the scene because the ambulance is running a Covid patient to the hospital.

And a lot of people thinking it was an overreaction proves nothing. People don't get a vote on reality.

[jrs235]

Honest question, are we better off in the long run, and is it a better solution, to decentralize energy generation and make more smaller grids rather than linking them all up? This isn't to say completely getting rid of the ability to transfer between the smaller grids to assist with power disruptions but to decouple and make it less likely for catastrophic "global" failures like this.

[mhandley]

With a high fraction of renewables, the reverse is probably better in the long run. The larger geographic area you connect, the less you're affected by weather systems, and the wider area you can draw dependable dispatchable power such as hydro from. But that depends on having enough grid capacity to move enough power around, which is currently a problem.

But I wonder from a reliability (or lack of cascading failures) point of view whether synchronous islands interconnected with DC interconnects is more robust than a large synchronous network?

[zanellato19]

It's hard to build big generators, so as we already struggle with infrastructure I don't think that's feasible, but it would be great.

[miki123211]

We're slowly reaching this point with the internet too.

I feel like to many technologists, the internet is still "the place you go to to play games and chat with friends", just like it was 20 years ago. Even if our brains know it isn't true, our hearts still feel that way.

I sometimes feel like the countries cutting off internet access during high school final exams have a point. If you know the internet will be off and on a few days a year, your systems will be designed accordingly, and if anything breaks, you'll notice quickly and during a low-stakes situation.

[pyrale]

Burying lines is not a panacea, it generates massive reactance changes compared to a classic line.

[yibg]

Maybe a good reason (in parts of the world where this is practical) to have some solar + battery storage. Doesn't even need to fully replace grid power, just enough to run the barebones when the grid goes out.

[marcosdumay]

If we are talking about a 72 hours blackout, I would be concerned about water and sewer working before one even thinks about home electricity.

[Vilian]

Houses have water tank that work as buffer for when the water stop, couldn't residential batteries work the same way?, it could detect drops in voltage and stop charging, so even if the grid is unstable in a black out, the residential isn't going to light up immediately, or only the houses without batteries