Hacker News new | ask | show | jobs
by josaka 1667 days ago
After the TX grid failure last winter, it's probably not a coincidence that Samsung's new facility will be near ERCOT's operation center in Taylor, which manages the TX grid, and will likely be the last load to shed when the grid's stressed. Used to work in the Austin fab, and the amount of money lost per minute in a power failure is mind boggling. The tax breaks Taylor offered ($314m) are not that different from what Samsung was reported to have lost due to the grid failure ($270m).
13 comments

The ERCOT campus has their own generation capability along with sitting between two distribution grids. Same with their DR site in Bastrop. In the case of a black start condition (grid totally down), ERCOT would need to come up first to coordinate the grid restoration.

Taylor is an ideal place for a large manufacturing operation since it's close to major highways and railroads, not too far from Austin, and land is very cheap.

> land is very cheap

*was

I'm between Taylor and Austin (Bagdad Rd, Leander). 1 acre lots along this strip have gone from ~$115k a year ago, to some recent sales at ~325k.

This...but did Samsung quietly negotiate a "any further energy supply f*ck-ups are 100% on YOUR dime" deal with the State of Texas, or what? By several accounts, the TX grid got darn close to collapse last winter - at which point "ERCOT Op Center is next door, and we get top priority" would not get you a single erg.
The going deal is, if there’s a power failure, we’ll let you recoup your costs from the taxpayers even as you profiteer. The governor can be bought for a very affordable $1 million. Certainly the incentives are aligned to solve this problem! https://www.texasobserver.org/after-kelcy-warrens-energy-tra...
If the costs do get passed to the taxpayer, let's hope the populace elects representatives who openly rip up the old contract and say "we're not paying". Flex the fickle power of democracy on the investors.
Also. Property tax in Texas is high relative to other states. So it makes sense to give Samsung a break on Ad Valorum taxes (not that a corporation with sufficient lawyers would pay them anyway.) They'll just make up the difference by stiffing it to the owners of all the new houses near the plant.

I can't imagine Samsung would put a plant in a 3rd world country like Texas unless they were getting significant tax deferrals from the state.

It remains to be seen if shifting the cost to taxpayers is harmful to the economy or not.
The ONLY reason to care about the "economy" is the secondary benefit it provides the "taxpayers". The health of economy and the people should never be weighed against each other.
Sure, lets analyze who benefits. The US benefits from national security by having a semiconductor plant in the US. The local economy benefits in Taylor Texas. Companies benefit from having a close semiconductor plant. People benefit from low cost chips. I think you may be advocating for more welfare analysis, which I completely agree with.
...but it will almost certainly hurt the taxpayers as a whole.
Not necessarily. Let's review our options: regulate companies to account for every disaster scenario, let the government plan and provide assistance during disaster scenarios, or do nothing and let people suffer and let companies go bankrupt. Passing costs to the taxpayers isn't the worst option.
Bad logic. If the governance e.g., TA is not manipulating market e.g. giving this sort of subsidies in case of failure then market could find the right place. Instead of behind the scene talks, a measurable study would be done and then fabs would be finding a suitable place to establish. I am not saying Samsung has not done studies on the location. Just pointing out the fallacy in the dangerous logic.
Letting companies go bankrupt is a good thing. Letting people suffer without power is not.
Crony capitalism, the best kind of capitalism eh?
Hacker news comment of the year
> erg

For anyone wondering, this is a unit of energy (100 nJ):

* https://en.wikipedia.org/wiki/Erg

(For those wondering, it's a unit from the old cgs system, rather than the mks we typically use today.)
FWIW, the problem wasn't a generalized grid management problem. It was a failure to properly winterize gas plants for rare freezing events. The Texas grid itself is in reasonable shape as far as I know.
Also, as I understand it, there's another "grid" which failed, which is the network of natural gas wells and pipelines. Natural gas wasn't being pulled out of the ground and pumped to the electric generation plants.

Apparently natural gas isn't easy to store, so basically there's not a lot of buffer, and everything needs to stay online or there won't be any natural gas to burn at power plants even if they were winterized, which some of them were.

I think some power plants may have even just shut down because natural gas was available but too expensive, and they weren't under a legal obligation to keep producing electricity.

Point being, to solve the problem, winterization of plants is just part of the problem.

The natural gas load was nearly double it's average load, because the entire state of Texas was below freezing for a week solid and almost nobody has gas heat in Texas. Everyone was pulling on inefficient electric heaters to warm their house. And the Windmills, which power about 22% of the Texas grid, went almost entirely offline.
“Almost nobody has gas heat in Texas” That’s incorrect. Approximately a third have gas heat per https://www.eia.gov/todayinenergy/detail.php?id=47116
Well, you must be a fun conversationalist. Nearly 2/3 of Texans use electricity for heat, and it no doubt becomes more electrically dependent in the southern latitudes of Texas (it being a very large and widespread state, climate varies quite a bit in various parts of the state).

By your own source that's 50% higher electricity dependence for heat than the U.S. average.

But no, I was not talking about the upper latitudes of Texas where freezing weather and snow is quite common and gas heaters are more prevalent.

Natural gas plant outages were around 30,000mw, and there were issues with both coal and nuclear plants going offline also. Wind was only a few thousand MW below planned values.

Not close to being entirely a wind power issue. (Even though wind power could clearly have done better, as it did in Iowa, where it was significantly colder and the wind turbines stayed online.)

https://www.statesman.com/story/news/2021/02/14/historic-win...

Over half the turbines were completely frozen. Wind typically accounts for ~22% of Texas power supply. Meanwhile, there was a record high spike (nearly double normal loads) in demand for electricity. If not for natural gas, the blackouts would have been even worse, because at least they were able to be ramped up and provide some power via rolling blackouts.

The wind turbines need to be winterized, too.

They didn’t do as well as they should have, but they did a lot better than expected.

> And the Windmills, which power about 22% of the Texas grid, went almost entirely offline.

Which was also a problem of not winterizing.

The picture of the entire event is that the government of Texas flagrantly ignored what it was told was the problem from the previous blackout, and then naturally the exact same problem caused the next one.

This week-long freezing event had never happened before in the state of Texas. The entire state was blanketed and plunged to temperatures colder than Anchorage, Alaska. We had single digit temps in Austin. Austin, has the same latitude as Jacksonville, Florida. This isn't some "flagrant ignorance" this never happened in modern meteorological history in Texas.
You might also want to read the analysis here around power sources during the outage: https://www.texastribune.org/2021/04/28/texas-power-outage-w...
No, the windmills actually produced more power during that cold period than was projected that they would be capable of. They didn’t go offline. A standard Republican refrain is to blame alternative sources of power generation for the faults that actually exist primarily in the old systems that bought off said Republicans. It’s like the pot calling the kettle black, but the pot actually is black and the kettle isn’t.

No, it was the gas-fired power plants that didn’t have enough gas to burn, caused by the fact that many of the gas producers had not winterized their systems, and in fact some of the gas producers got their own power turned off in the rolling blackouts — they hadn’t been marked as mission critical.

Try reading the report.

Sadly, very little has changed since February. The producers and the gas power plants have paid a trivial fee that allows them to choose not to winterize (because that would be too expensive), and so far as I know they still haven’t even bothered to mark those systems as mission critical, so when the rolling blackouts happen again, they’ll get their power cut off again.

For anyone interested, FERC basically just said this as well in their report on the outage.

https://www.ferc.gov/media/february-2021-cold-weather-outage...

Of course they said a bunch of things and made a bunch of recommendations the last time ERCOT fucked up, and they didn't listen to nearly any of them, so I would expect the same this time.

"The grid" tends to refer to more than just the transmission wires.
That seems unnecessarily sarcastic. I wasn't saying just the "wires". I was trying to shed more light on what the actual problem was.
Tbf, lots of people conflate transmission wires with "the grid". Even on a fairly tech-savvy forum like HN, I've read where people insinuate things like if all ICE vehicles were suddenly replaced with EV, all "the grid" would need is more wires running to charging stations.

Maybe people consider the grid distinct from all the supporting infrastructure.

“The grid” never refers to just the transmission wires.
I was surprised the power outage affected Samsung because in DFW the outage only affected residential areas. I didn't see a single commercial/industrial space without power including fully lit empty parking lots.
What finance guy will ever approve this? I doubt that the service agreement would ever be approved with that clause.
LOL, the current political heads for the state of Texas (and county of Taylor) will happily trade future liability of their constituents for investment now (and kickbacks later). Honestly, I think Samsung's management is more comfortable in this sort of "political" environment than many others.

If you watch local news reporting, it's hilarious that they (politicians) have no idea what the "factory" does. I'm personally highly doubtful that a fab which hasn't broken ground will be producing viable 5nm at the 2H of 2024 as claimed (or cyber trucks for that matter).

> the current political heads for the state of Texas (and county of Taylor) will happily trade future liability of their constituents for investment now (and kickbacks later)

Do you have some examples of this?

ERCOT, the massive statewide blackouts, and bailing out massive price gouging of the electric utilities that basically makes the reliability of the grid an issue in the first place? https://www.texastribune.org/2021/05/25/electricity-market-f...

Or, in general: https://www.texastribune.org/2017/08/09/are-business-ties-be...

Billion dollar "public/private" investments turn in to boondoggles except for those who retire as lobbyists or foreign firms who pick up the pieces of the bankruptcy.

You might want to do some research into the people in charge. You're kind of implying that those people would listen to bean counters over their pollsters. This is where the flaw in your logic is located. Also, you're using logic in the first place ;-)
Surely there's an insurance company that would take on that risk, and the government could pay for it. These government incentives are rife with excessive spending on a per-job/total economic impact basis. I see no reason why this detail would derail things.
Location close to ERCOT is not the reason.

Samsung for years has had a major facility outside of Austin.

Samsung Austin Semiconductor https://goo.gl/maps/nX9BFxHoVMCVKBLL7

And ERCOT is also based outside of Austin.

Austin is also the state capital, centrally located & home of one of the nations largest universities (UT), who has a strong engineering & CS program graduating thousands of students per year.

So it shouldn’t be a surprise that major companies have a significant presence in or around Austin.

I really doubt proximity to the ERCOT center has anything to do with it. The new facility isn't close enough at all to the ERCOT center to be on the same grid segment. And with a new facility like this that is in the middle of a field outside of town, the new fab will be built on its own grid segment anyway, completely separate from the rest of Taylor and the ERCOT center.

Taylor was chosen because that area (northeast of Austin, just south of Taylor) is one of the last areas near Austin that has abundant amounts of wide open fields ready to be developed. They also got a good deal on the water usage, and of course the tax breaks from Williamson County.

Do you have any thoughts on how water availability will be in the years/decades to come and how a fab would impact it and the communities depending on it?

My understanding is when you cross east past round rock towards Hutto/Taylor, the water aquifer being used changes to use water piped over from Carrizo-Wilcox aquifer.

Newer fabs like this recycle a huge portion of their water. I did some back of the napkin math a few weeks ago and the amount of water that the fab will be drawing (after accounting for the recycling) will be about the same as the water usage of an equivalently-sized suburban neighborhood. So while water usage is definitely a concern, I'm not sure that this fab is a bigger concern than your typical neighborhoods or farms.
tangentially related bit of trivium... when i was house hunting a looked at a house across from a fire station, and upon doing research about what it's like to live across from one, found that a.) they tend not to blast sirens upon exiting the station, in contrast to living near a hospital with an ED, and b.) the power grid segments that fire stations are on have priority and are less likely to lose power during blackouts.
I wonder if the siren thing varies from location to location, a good friend of mine lived right next to one and it seemed you couldn't go 15 minutes without being drown out by them heading out.
I wonder if the siren thing varies from location to location,

In my experience (former firefighter here), it absolutely does. It also varies by call type, time of day, etc. So if we got a call at 2:00 AM and there was no traffic on the road near the station, we might leave the station with no siren out of respect for the people trying to sleep in the homes very close to the station. This would be more true if the call was a lower priority call in the first place. OTOH though, if the call as "residential structure fire with occupants reported trapped" our focus would not be on sleeping neighbors and there's a better chance that we'd be getting on the Q pretty much right out the door.

Another factor for us was that our station was right next to an intersection that was very busy at times, and known for many traffic accidents. So any call where we had to turn left (towards the intersection) out of the parking lot, there was a better chance we'd be hitting the siren and air-horn pretty much right from the jump.

Anyway, yeah, net-net, this is going to vary based on lots of factors: department policy / culture, geography, time, traffic, etc, yadda yadda.

>getting on the Q pretty much right out the door

Q?

My office is adjacent to a fire station. The policy has changed over the years. They used to use the sirens all the time. Now they almost never use the sirens when they leave. In a way that makes it more disturbing when they do, you know there's a reason.

Other things which are bothersome about a firehouse…

• They put the communications radio on some sort of loud speaker sometimes, so you get to listen to all the calls in the city.

• They have some sort of turbine that runs a lot. Maybe a hose dryer? I haven't gone to ask, but when I used to do audio recordings there were days we couldn't record because of it.

• Their backup generator optimizes efficiency by minimizing the muffler. The tests are pretty loud.

• The building department made me block up the windows on their side because I was too close to the property line and a fire might spread to the fire station. I say if the fire department can't keep fire from coming out my window and setting fire to their building 60 feet away, that's at least partly their fault. :-)

• They are a multi-tier bureaucracy. When their tree slowly fell over on to my building it took many months to get them to admit it was their tree and maybe they should cut it down. Lots of finger pointing between the fire department and various city levels and a few "dead parrot" conversations with different functionaries.

If the fire station is located on a busy road or near a blind curve, turning on the siren when exiting might be a good policy. Out on a county road with good sight lines and low traffic? They can wait until they're going at a hazardous speed or intersection.

My main concern would be all of the communications equipment they typically have. Being that close to high powered transmitters can cause annoying interference.

Not just a good policy -- my wife is a former EMT & firefighter in MN, and at least there, apparently it's the law that fire trucks blast their horns at every intersection. Whether this is required when they leave the fire hall and turn onto the first road, I don't know.
A conversation may yield better too. If they know they are waking a day sleeper, for example.

I know I would attempt one. Try nice and some charm and see what comes of it. May end up popular with neighbors.

He's since moved, I'm not sure if he had spoken to them, but I think you're right that it would have made him a hero of the area. This was a pretty densely residential area of Waukesha, WI - the fire department near the airport.
Well, maybe a passerby benefits from this exchange.
In NJ they go off all the time to alert the volunteers not at the station, and it's used for all emergencies, not just fires that need trucks to roll out.
I live in NJ, directly across the street from a fire station. I also occasionally work nights to connect with team mates in other time zones.

You will always see the flashing lights but they have never blasted the sirens at night.

I lived in an area that lost power every time there was a stiff breeze. This was back in the 1990s, when we still used telephone modems. I blew up 3 of them. 2, even with surge protectors.

But power came back quickly, because I was on the same trunk as the fire station.

I used to live a mile or two away from a hospital and getting the lights back on before everyone else was nice.
> the power grid segments that fire stations are on have priority and are less likely to lose power during blackouts

So do large grocery stores, for obvious reasons.

It has nothing to do with ERCOT presence, despite the social karma of referring to the “grid disaster.”

There have been literally hundreds of other grid disasters in New York, Chicago, the entire state of California, and yet no one would be surprised if the plant showed up there. Why?

Isn't Samsung big enough to simply ask permissions to build their own nuclear reactor at that point ? Or wind turbine farms or whatever ?
A new nuclear plant cost billions and takes years to get permission and construction to complete. I assume they rather want to make chips, soon.
Well, for the sake of entertainment let's see: I suppose Samsung still want to be able to manufacture products in its Texan factories for more than 5 years (which according to my small European contry is roughly the time it took to build a nuclear power plant in the 70's) and they have billions (oh and Apple too) and according to Google's first results:

     Companies that are planning new nuclear units are currently indicating that the total costs (including escalation and financing costs) will be in the range of $5,500/kW to $8,100/kW or between $6 billion and $9 billion for each 1,100 MW plant.
So why not ? They'd control one more link in the supply chain.
"So why not ? They'd control one more link in the supply chain."

Because operating a nuclear reactor is a whole different thing, than operating a chip plant and you usually want to focus your energy, not spread it out. And when the goal is, to just have blackout save energy for your chip plant, then there are way cheaper, riskfree and faster solutions, like batteries, or gas generators.

Reactors and wind farms still need a grid.
Not if they are only interested in getting power to their factories ^^.
They are setting up fairly close to Tesla. Wonder if they are working on a deal for Tesla to provide battery backup. Interestingly enough Austin, TX appears to be rapidly approaching SF status as a tech hub and may soon surpass them. Its an interesting thing to watch and really gives insight into how politics and taxes affect business and growth.
> Wonder if they are working on a deal for Tesla to provide battery backup.

Running an entire leading edge semiconductor plant on battery backup is a crazy proposition in my view. Especially, for intentions of surviving another Texas winter situation without any disruption to operations. Approximately 24 hours of full-coverage battery backup would have been required to bridge the rolling blackouts.

Last I checked (about a decade ago), the SAS A2/S2 lines accounted for ~13% of the electrical load for the entire city of Austin. Just look at the amount of power one EUV light source requires today. These lines will have potentially dozens moving forward. Then consider that all of the photo area accounts for a tiny fraction of total power consumption in one of these plants.

If it were even remotely feasible for the Samsung semiconductor lines to operate on standby generator power, they would have installed these units already and no losses would have been incurred.

They might as well be smelting aluminum behind those walls. Any backup generator power or UPS devices are designed for life safety and stopping the line without causing 10+ figure losses. You cannot run one of these facilities on in-house power. Another commenter proposed a nuclear reactor installation. This is actually not a terrible idea once you understand the scale. Putting 10-15 semiconductor lines around a nuke plant makes perfect sense to me.

The Bay Area had something like 20x as much VC investment as Austin as of 2017 [0]. This article [1] indicates that the trend is probably in Austin's favor, as I'd expect, but it also shows that Austin isn't even in the top 10 nationally (unless they lump it in with San Antonio?) by deal count as of 2020.

[0] https://pitchbook.com/news/articles/the-bay-area-beyond-rank...

[1] https://www.cnbc.com/2021/01/14/silicon-valleys-share-of-ven...

> Interestingly enough Austin, TX appears to be rapidly approaching SF status as a tech hub and may soon surpass them.

Uh, maybe it's time to take a drive through Austin, but that was a pretty sleepy little college town 10 years ago. Most of my family lives in the central Texas region (San Antonio, San Carlos, College Station), and I presently live in Mountain View. I have a hard time believing SF-scale infrastructure was built that fast.

As an example, here's the UT wind tunnel: http://research.ae.utexas.edu/floimlab/Mach5.php

Here's the Moffett wind tunnel complex: https://www.nasa.gov/centers/ames/multimedia/images/2005/nfa...

+1 for comparing wind tunnel capacity. Also TX has no dirigible hangars.
Can't stop laughing
Companies that require wind tunnels are no longer “tech companies” and are therefore irrelevant for a discussion about “tech hubs”.
Yeah but San Antonio's even better, here's their car factory: https://s.hdnux.com/photos/46/41/53/10099417/69/1200x0.jpg
I think he may have meant in terms of software companies. Many of them do not require a wind tunnel.
I primarily meant software, computer manufacturing (chips, etc. ) style jobs but the fact that you went with wind tunnels is awesome :) You will always be cool in my books for this comment.
I don't think wind tunnels (of which these seem like fundamentally different types) are a great example for this argument.
I don't know about SV status but I think it's getting there. Other tech hubs still extract graduates out of the state at an alarming rate and last I checked compensation wasn't close to SV (even with normalization for cost of living). It'll take a good bit of time to undo SV but the ball is rolling.
care to share any sources? or did you just make this up?
Make what up, The Tesla link? I just asked the question, I have no info either way, notice I said "Wonder if".
> rapidly approaching sf status. That should be a quantifiable metric?
Austin has seen a 40+ % increase in tech jobs in a decade and has recently attracted major companies like Tesla and oracle to headquarter there while the majority of FAANG companies are building large campuses there.

This site does a pretty good job: https://sfciti.org/sf-tech-exodus/

Article from the SF Chronicle: https://www.sfchronicle.com/bayarea/article/Austin-COVID-tec...

Hopefully that addresses your question as to why I hold this opinion.

Yes, they made it up

According to wikipedia/some google-fu SV has just above double the high tech jobs as Austin

That would be way more high tech jobs in Austin per capita.
According to wikipedia/some google-fu Silicon Valley has a population of ~ 3 million people while Austin has a population of just under a million. Your numbers support my comment that you say I "Just made up"
I'm a big fan of Tesla and Mr. Musk - but does anyone know of any non-vaporware large scale industrial facilities that use Tesla battery walls as power backup? Do Tesla manufacturing plants do this?
Not sure about industrial, but they have grid scale battery storage with their Megapack product. Hornsdale in Australia has been operating for a few years. https://www.greentechmedia.com/articles/read/tesla-fulfills-...
https://www.bloomberg.com/news/features/2021-03-08/tesla-is-...

Not quite what you were asking about but it does deal with the TX grid.

What are some interesting startups in Austin area?
I doubt it. The land is just cheap and widely available over there. Also, they know workers will be attracted to being a short commute from Round Rock. The storm that caused the grid failure is a very rare occurrence ("100-year storm").
Which means that it will probably happen this year, too. And every year after this, thanks to the global catastrophic weather changes.
Doesn't it cost less than that amount of money to have on-site backup power?
If you are that big of a deal, wouldn't you have your own power plant with reserve power? My university had 2 power plants. Surely it isn't that hard
As more and more companies start to invest in Texas there is more demand in tigetening the grid to avoid future catastrophes.
Why not use generators like most data centers in case of power failure? Is energy usage that much higher?