[_heimdall]

If and when these tools become self-improving we have absolutely no idea what comes next.

Maybe its utopia, maybe its akin to an Eliezer Yudkowsky prediction, who knows. Regardless of the specific outcome, its a huge gamble with effectively unlimited risk.

[cwmma]

The architecture of the current models where learning is a separate and very expensive process make runaway self improvement seem like something that will require a bunch of breakthroughs in order to happen.

[_heimdall]

That seems reasonable, but I'm not sure if we really know yet. If current models are given direction and control to change how the next model is trained or architected it seems plausible that they could stumble into such a breakthrough.

The current LLM approach makes huge assumptions, including that training only on text prediction is enough to simulate true intelligence. That may or may not be a valid assumption, but it could be enough for the LLM to make one seemingly small change that ends up running away from us faster than we would realize.

[tim333]

I doubt there will be a sudden run away. Probably more of a gradual shift where humans and AI combine to improve the AI and gradually the AI does more of the work.

[_heimdall]

I'd agree that seems more likely. Though I can only make that expectation with public knowledge of what these AI companies say they're working on. I don't know what research projects or yet-to-be-released architectures may currently exist - my confidence in that assumption of likelihood isn't very high.

More importantly, I think we agree a sudden run away is possible, we don't have safeguards in place, and the risks are extremely high should that happen.