Hacker News new | ask | show | jobs
by gregimba 1945 days ago
Currently in ATX we haven't had power for over 24 hours, wood fireplace is saving the house from completely freezing but the whole city of Austin needs to evaluate how they setup their critical infrastructure to allow for rolling blackouts instead of having parts of the city with power and parts without for the entirety of an outage.
4 comments

Rolling outages work when you need to shed 1% of your load, not when you need to shed 80% of it. If Austin is down to critical loads only (hospitals or whatever) there may not be anything left to rotate.
There are non-critical loads all over that were never interrupted (my neighborhood, north edge of ATX). It's pretty arbitrary.

Even worse: for about the first 24h there were (largely uninhabited!) commercial buildings all over the place that were fully lit up. We're talking skyscrapers. Even some unfinished skyscrapers. I think they've finally begun to address those. The charitable interpretation is that it was simply a massive failure of coordination.

Do grid operators really have controls at that level? In California they can shut off a substation or not, and that's as low as they can go except in the special case of industrial customers with demand-responsive equipment installed. If those empty skyscrapers were on the same local circuits as a hospital, perhaps it simply wasn't possible for the electric company to turn them off.
Around 2010 or so I worked for a skyscraper in downtown Seattle. There was a heat wave and the utility needed to shed load. The mechanism for doing that was someone calling our front desk and saying "Hello, please shut off your lights."

(Commercial lighting runs at 277v, so it's all on separate circuits from wall outlets. You can shut off the lights in a building without killing the servers, for example)

We had remote-controlled breakers, so doing that was a couple clicks of the mouse. But if nobody had picked up the phone, they would have needed cops to break into the electrical room on each floor and start flipping breakers by hand.

It's more that they needed to communicate with the owners of the buildings to shut off massive systems that weren't even being used, while the minority of residents who still have power are being asked to "live as if we didn't" and navigate by candlelight, etc.
If the grid operator has deployed modern Smart Meters to their customers, they can control and measure electricity to the house level remotely.
Keep in mind: the details of how generators shed load is not ERCOT's decree. If there is unfairness in how liad shedding is taking place, you need to look at the entities generating/transmitting it. Not ERCOT.

Bluebonnet has done a great, if annoying job. They converged to a near 50/50 duty cycle I think between two trunk lines.

I've heard Austin Energy is epically failing some of it's customers though.

The people getting dangerously cold in their own homes don't care about the nuances between ERCOT and Austin Energy and their city government and county and state and federal governments and this private organization and that. The system as a whole has failed. In tragic fashion. Period. Modularize your organizations if you want to, but it is not an excuse for passing the buck.

There are people at the top of the whole pile. And they have the authority and the responsibility to make sure the whole thing works, at the end of the day, no matter the implementation details. And the whole thing doesn't work.

This is what the American system is chronically worst at. We delegate, and we contract out, and we federalize, and we privatize, and we divide responsibilities. We avoid centralizing things at all costs. And then when put under pressure, those separate pieces often fall apart. Communication fails. At best nobody knows what's going on, at worst they willfully ignore responsibility because somebody else will end up with some or all of the blame. This keeps happening over, and over, and over again. I can't help but feel our society is crumbling.

>The people getting dangerously cold in their own homes don't care about the nuances between ERCOT and Austin Energy and their city government and county and state and federal governments and this private organization and that. The system as a whole has failed. In tragic fashion.

Mayhaps if they did, they'd have seen warning signs that such an eventuality was inevitable in coming as they'd have a firm grasp of who was responsible for what, and had their hope of someone getting it just right wiped from their minds and replaced by the grim fact that the best tool they could employ to their own survival is that matter within their own head.

>Modularize your organizations if you want to, but it is not an excuse for passing the buck.

Part of Modularization is clearly defining and delineating roles and responsibilities. ERCOT's is to be the tracker and issuer of EEA's. That means having the authority to instigate, not implement, instigate, rolling blackouts. Make it happen; not how, just that it needed to. It isn't ERCOT's business other than to keep everybody dancing to the sane tune, and to keep track of the numbers.

Each provider went and did that; Some to great success. Even mine. I, in fact had to make some extra clever use of those times I had power to put it to the best use to stabilize the situation in our household.

>There are people at the top of the whole pile.

Funny thing about being on top of a pile, you're just as clueless as to what's actually on the bottom unless you actively go look into it, which is a calculated tradeoff that may distract you from doing something only you can see to do from where you are.

>And they have the authority and the responsibility to make sure the whole thing works, at the end of the day, no matter the implementation details

Oh, you sweet summer child. You think it's just a case of hup, two, three, four, and there you go, ERCOT makes your problem go away?

ERCOT owns nothing. It's a platform. A glorified clearinghouse. A market in which a bunch of private entities sell their wares, in this case, generation of power, usage of transmission infrastructure, etc.

There's no authority to magically make it all work. There's process, a whole lotta tooling, hopefully a pretty good chunk of people smart enough to use it well and sensibly, and a common agreement as to who has final say. In ERCOT's case, that jurisdiction and authority is well defined, and limited in scope.

>This is what the American system is chronically worst at. We delegate, and we contract out, and we federalize, and we divide responsibilities, and when put under pressure those pieces tend to fall apart.

Welcome to the real world. Where people like me, and now you too have come to the epiphany that there's some level of "inevitable failure" at play because companies act like bored people more than happy to pass the buck, and are fundamentally flawed, collective creations, of implicitly flawed beings. Mistakes will always happen, as will miscommunication. We contract out in good faith, it isn't always recipricated perfectly. We delegate, and we have to accept what we get back even if it only sorta marginally resembles what it was we asked for. We federalize, generally to make some common thing formally a common thing, but we also open ourselves to abuse by doing so.

There is a solution though. That's for people to get dead serious about doing damn good business. Something which can't happen in an environment of natural monopolies or industrial monoliths, a concept enshrined in the architecture of the Texas Grid, and enshrined in American system as a whole, though you have to rip off a few decades here and there of astonishingly bad ideas that are best characterized as cranial rectal insertions to see it.

Competition -> innovation -> newfound possibilities propagate through the competitive environment -> repeat

The last thing anyone needs is more conglomeration, if anything we need more people cranking on the same problems, cross checking everyone else to figure out if anything has been missed, and to ensure there is enough overall fault tolerance in the system.

Texas is, and will remain, what it is. We rebuild, but better. We try, and make it work as best we can. You may not like it, but a not inconsequent number of them do.

As an Austin Energy customer a large portion of the blame for this on their poor infrastructure configuration and the number of critical circuits.
Unfortunately Austin has not done a very good job of designing it's grid. There are many sectors which can't be rotated as they contain one or more critical services, even as they also contain many consumers who are not critical. This means that instead of being able to rotate amoungst the ~80% of non-critical power users, they are only able to shut down about 45% of the power.

Unfortunately in a situation like this that means that 45% have remained off for the past 48 hours (no rotation is possible), while many empty buildings are fully illuminated. It is a technological failure of their ability to shut down specific power users.

From what I hear, only about 40% are on non-critical circuits. There is room for improvement.
This is essentially the crux of the problem, multiple friends/and coworkers are staying in islands of power that have been interrupt free due to proximity to Fire/Police/Hospitals and others have been without power for extended periods of time, if 60% of a grid is essential you can't roll non essential loads because simply maintaining the critical loads takes up 100% of capacity. More granular control could significantly help alleviate this issue which is specific to Austin Energy.
True crtical loads have backup generators up to this task, or the maintenance people should be fired for incompetence. Though they could get a break for some failures of the system, but only a handful statewide
I had my wood fireplace going for the first time this weekend when the daytime temperatures were around -12F. It can easily raise the temperature of my living room, dining room & kitchen (one open space) to 80+ but that's with electricity to power the circulation fan.

How are you keeping the whole house warm if you can't circulate the air? I'm genuinely curious.

There's some old school technology to solve that problem. For example, I stayed in a friend's ancient family home in Maine that simply had a bedroom over the living room wood stove, and an open vent between the two floors to let the heat through. There are also non-electric fans to help distribute heat from a wood stove.

https://www.amazon.com/PYBBO-Improved-Fireplace-Magnetic-The...

To clarify, that fan is electric and powered by a peltier (thermoelectric effect) device using heat from the stove.
This is exactly why old farmhouses here in the Midwest were T shaped. There was a chimney in the center of each long arm of the T. Then vents through the floor right above the heaters to warm the second story. With good placement they did not need to blow the heat around.
From personal experience, a heat-powered fan helps a lot. I used one when I lived in a Franklin-stove-heated cabin in Montana, and while you wouldn't mistake its power for an electric fan, it more than did the job.

https://www.amazon.com/heat-powered-stove-fan/s?k=heat+power...

Currently running a generator to run the wood stoves fan, its not ideal but not amazing.
car battery and computer fan
SAT going on 24 and 36 hours of effectively no power. It's 44 in room.
ATX?
Austin Totally Xtreme?
Yes. Beyond a doubt, yes.
Austin, TX