[mrothroc]

Let me ask you this: would people accept a nuclear power plant whose safety control software is vibe coded? Would a CFO of a public company sign off on financial statements created entirely by AI?

I'm fully into agentic coding, and I'm actively studying agent reliability. I can make all kinds of deterministic guarantees about agentically produced code, but no, I would not accept either of these, or many other examples.

We write code (or we used to, before AI), so we naturally value that. But the code is one small part of a deployed system, and this has always been the case. Numerous studies have shown that writing the code is actually the cheapest part.

All the most important things that you need to know about what makes the code correct, both before it is written and after it is deployed, are not in the specs. They come from walking around and talking to people. Looking them in the eye, sussing out what their real requirements are, and figuring out how to address their concerns with empathy.

Until LLMs can do that, I'm not worried. Let them write the code, that's the least important part.

[atleastoptimal]

> They come from walking around and talking to people. Looking them in the eye, sussing out what their real requirements are, and figuring out how to address their concerns with empathy.

This claim depends on humans remaining the bottleneck to high-leverage information, necessitating a human<->human interfacing role that solicits requirements, ascertains intent, etc.

I don't deny that that is very important and cannot be done by AI now. However, my concern is that AI will be much better at any domain of information processing, and organizations that gate important decisions being made by a network of barriers and information silos dependent on "talking to people" will be outcompeted by largely autonomously run AI agent organizations, which have, by their very design, far higher throughput, auditing, memory, parallelization, etc.

It's kind of like saying that machines could never make fabric because it is impossible for a robot to replicate the complicated motion of a human threading a needle. The industrial revolution was prompted by creating machines which redesigned the entire process to account for machine limitations, and allowed the superior speed and scale of machines to drive higher productivity, delegating humans to a role of maintenance and simply feeding the machines their needed input.

[mrothroc]

Yes, at some point AI will be fully integrated into society so that there are entire autonomous sections. But society doesn't move at the same pace as technology.

The output of agents has to have economic value, and for the foreseeable future this means someone is going to have to buy something.

Right now, it is humans who ultimately make the economic decision. Even if you have fully autonomous agentic organizations selling to each other, there are one or more humans at the end of the of the chain who agree to exchange money for value.

Science Fiction writers have envisioned futures where some currency other than money is used to track value, but so far as I can tell we are nowhere near moving to anything like that.