[Topfi]

My personal issue in comparing LLM progress and risk as labs publicly predict it with nuclear power in the middle of the 20th century is that the processes by which it works where fairly quickly well understood and the risk could thus be realistically assessed. Some powerplant operators did not adhere with best practices, but building a relatively safe nuclear power plant was not impossible given appropriate effort and spending. Heck, according to some, we could have even gone far more fail-safe approaches (molten salt) if military interest haden’t been at play.

With what is predicted by frontier labs for LLMs, all of this is not the case. We are far further from any understanding of how these models work internally than in the early days of fission and, if this was actually creating a truly intelligent, autonomous entity, alignment seems unsolvable as well, at least the way it is proposed.

It’s why I have from the get go been critical of this doomsday framing and tended to always dislike it. This is basically the outcome that was inevitable given the framing and it was bought to prevent far less stringent, but more actionable possible regulation that labs very much wanted to avoid.

[SXX]

  > We are far further from any understanding of how these models work internally than in the early days of fission

OMG. I'm like really dont want to be offensive or something, but everyone always knew "HOW" these models work exactly. Its easy enough principle to explain to 10 years old if you take something like Karpathy article on MicroGPT:

https://karpathy.github.io/2026/02/12/microgpt/

None of SOTA LLMs are any different - they just much much larger and have a lot of optimizations.

Fact that LLM companies trying to sell it as some kind of magic is just proof how much lies is here.

All it does is just predict next "word" at any given time.

  > and, if this was actually creating a truly intelligent, autonomous entity, alignment seems unsolvable as well, at least the way it is proposed.

This is obviously true. It's very hard to predict whatever you gonna decompress from a lossely "compressed" dataset using floating point math.

This is why you cant solve it all with pre-training or censorship on top, but instead you need a good sandboxes and harnesses.

[Topfi]

By how, I meant specifically the internal activations, which no person in the field claims to have a comprehensive understanding of, not next token prediction as the underlying technology. The whole interpretability of it all is the crux I was referring to, though I will give that you are right, that’s not really the how it works and I worded it sloppily.

Anthropic are putting more effort than most into this and I find their work fascinating in that area, though like with OpenAI, I will maintain that if they truly believed this problem must be solved to stave off major catastrophe, they’d solely focus on interpretability of other labs models, not work on and market their own.

[ToValueFunfetti]

All humans do is predict the next action at any given time. You roll your eyes, it's a tired argument, but still. You have memories, a personality, thoughts ranging from the long running to the mere reflexive, you have a rich conscious experience, and all of this in service of generating the next thing that you do at any given time. If you actually knew how LLMs worked, you could rewrite them as code, refactor it, disable jailbreaks, and put out a superior product. Your description only covers what an LLM does, not how. Part of the how is that it necessarily predicts multiple words ahead. It wouldn't be possible to write couplets otherwise, and they could do that in the GPT-3 era.