[whimsicalism]

I agree with the first part which is basically 'being able to do a software engineers full job' is basically ASI/AGI complete.

But I think it is certainly possible that we reach a point/plateau where everything is just 'english -> code' compilation but that 'vibe coding' compilation step is really really good.

[pron]

It's possible, but I don't see any reason to assume that it's more likely that machines will be able to code as well as working programmers yet not be able to come up with requirements or even ideas as well as working PMs. In fact, why not the opposite? I think that currently LLMs are better at writing general prose, offering advice etc.., than they are at writing code. They are better at knowing what people generally want than they are at solving complex logic puzzle that require many deduction steps. Once we're reduced to imagining what AI can and cannot do, we can imagine pretty much any capability or restriction we like. We can imagine something is possible, and we can just as well choose to imagine it's not possible. We're now in the realm of, literally, science fiction.

[whimsicalism]

> It's possible, but I don't see any reason to assume that it's more likely that machines will be able to code as well as working programmers yet not be able to come up with requirements or even ideas as well as working PMs.

Ideation at the working PM level, sure. I meant more hard technical ideation - ie. what gets us from 'not working humanoid robot' to 'humanoid robot' or 'what do we need to do to get a detection of a higgs boson', etc. etc. I think it is possible to imagine a world where 'english -> code' (for reasonably specific english) is solved but not that level of ideation. If that level of ideation is solved, then we have ASI.

[agentultra]

There are a ton of extremely Hard problems to solve there that we are not likely going to solve.

One: English is terribly non-prescriptive. Explaining an algorithm is incredibly laborious in spoken language and can contain many ambiguous errors. Try reading Euclid’s Elements. Or really any pre-algebra text and reproduce its results.

Fortunately there’s a solution to that. Formal languages.

Now LLMs can somewhat bridge that gap due to how frequently we write about code. But it’s a non-deterministic process and hallucinations are by design. There’s no escaping the fact that an LLM is making up the code it generates. There’s nothing inside the machine that is understanding what any of the data it’s manipulating means or how it affects the system it’s generating code for.

And it’s not even a tool.

Worse, we can’t actually ship the code that gets generated without a human appendage to the machine to take the fall for it if there are any mistakes in it.

If you’re trying to vibe code an operating system and have no idea what good OS design is or what good code for such a system looks like… you’re going to be a bad appendage for the clanker. If it could ship code on its own the corporate powers that be absolutely would fire all the vibe coders and you’d never work again.

Vibe coding is turning people into indentured corporate servants. The last mile delivery driver of code. Every input surveilled and scrutinized. Output is your responsibility and something you have little control over. You learn nothing when the LLM gives you the answer because you’ll forget it tomorrow. There’s no joy in it either because there is no challenge and no difficulty.

I think what pron is leading to is that there’s no need to imagine what these machines could potentially do. I think we should be looking at what they actually do, who they’re doing it to, and who benefits from it.

[jimbokun]

The only reason to imagine that plateau is because it’s painful to imagine a near future where humans have zero economic value.

[goatlover]

It's not the only reason, technologies do plateau. We're not living in orbiting cities flying fusion powered vehicles around, even though we built rockets and nuclear power more than half a century ago.

[pron]

Yes, but perhaps technology can't plateau beyond vibe coding but below "the machine does everything", not because technology doesn't plateau but because that point doesn't exist. Technology could plateau before both or after both.

[jimbokun]

All the current indicators are that AI will plateau far beyond any human capability.

Do you have evidence or empirical arguments to the contrary?

[FrontierProject]

Burden of proof is on you sir, not the other way around.

[jimbokun]

There is no proof for things that haven't happened yet.

I just find the belief that the continuous rapid improvements in machine intelligence stopping just short of human intelligence quite arbitrary.

[Anamon]

The definition of what we see progress in with LLMs as anything akin to intelligence is your supposition, though.

It's a view I don't share at all. I don't see anything that could be even remotely considered intelligent under any reasonable definition of the term. Useful in places, yes. Intelligence, hard no. So the question of whether it will approch or surpass human intelligence makes no sense to me, because the technology and its progress are in an entirely different dimension and unrelated to that idea or goal.

[tjr]

Why is this desirable?

[jimbokun]

It’s not, it’s horrifying.

But there doesn’t seem to be any off ramp, given the incentives of our current economic system.