[pjc50]

I think this is where aggregate effects have to be considered. One person building an idiosyncratic house out of found materials: neat little project. One million people doing so: shanty town that can be seen from orbit and is a disaster waiting to happen.

The Web already had a problem with externalizing costs onto users. Both the simple cost of poorly executing websites (power, mobile data, time), and more subtle ones (social media). AI is a huge accelerant for that.

[ethbr1]

Imho, this is the biggest concern for civilization-scale cost of AI being deployed all fields.

On the one hand: Amazing power! Look at how much more output everyone is generating!

On the other hand: Society maintaining guardrails that were engineered around human failure modes are completely inappropriate for bounding something that makes 10% random errors everywhere.

To re-use an example upthread... what if you had an LLM that could build an entire house? Foundation? Plumbing? Electrical? Roofing? Septic?

And then we let everyone guide that LLM to build their own house.

We'd drastically increase the housing supply, but every house would have 10% of its systems built incorrectly.

We could fix that by requiring grounding in building codes, automated reviews, etc. But none of that hard work updating guardrails is something AI accelerationists are interested in working on.

So you end up with a society where every septic system has a 10% chance of being misdesigned...

[27183]

> We could fix that by requiring grounding in building codes, automated reviews, etc.

It's an open question whether, with all those guardrails, AI would provide any speedup.

[ethbr1]

Absolutely. I have an inkling that there's an AI version of Amdahl's law to the effect of: the speedup obtained by AI is proportional to the amount of AI output that isn't verified.

Folks push back and say "But there are tons of things where deterministic verification is easy, but design is hard."

But I'm suspicious that in real world cases verification is a lot harder and more time-consuming (because you can't verify 95% of something and call it a day).

[27183]

I have a hunch that most of the things humans do are AI-complete. Very little of it can be easily automated directly without AGI. We can achieve automation by re-orienting our workflows around machines (factory assembly lines, for example), but that isn't always possible or useful. That's what makes ML applications difficult--finding how to re-orient a workflow s.t. an ML algorithm can be used productively in it.