| Imho, this is the biggest concern for civilization-scale cost of AI being deployed all fields. On the one hand: Amazing power! Look at how much more output everyone is generating! On the other hand: Society maintaining guardrails that were engineered around human failure modes are completely inappropriate for bounding something that makes 10% random errors everywhere. To re-use an example upthread... what if you had an LLM that could build an entire house? Foundation? Plumbing? Electrical? Roofing? Septic? And then we let everyone guide that LLM to build their own house. We'd drastically increase the housing supply, but every house would have 10% of its systems built incorrectly. We could fix that by requiring grounding in building codes, automated reviews, etc. But none of that hard work updating guardrails is something AI accelerationists are interested in working on. So you end up with a society where every septic system has a 10% chance of being misdesigned... |
It's an open question whether, with all those guardrails, AI would provide any speedup.