[27183]

> Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created.

Then why does anyone expect to create it? I'll take a stab at an answer: they think an LLM is some kind of "incremental improvement" and therefore a step along the inevitable path to discovering AI. But that seems delusional to me. I can't imagine anyone sound of mind who knows how an LLM works thinks it's actually intelligent. So in what sense is it an "advancement" on the path to AI?

The concept of an incremental improvement in an objectiveless search in a high dimensional space is.. absurd.

[kalkin]

> actually intelligent

It's reasonable to doubt that LLMs are a path to AGI, but I don't understand how this is still a matter of dispute in 2026. What's your definition of intelligence that doesn't cover an entity that can translate fluently between dozens of languages and also solve open problems in mathematics? And be real-if you have one, is it a definition you or anyone would have given a decade ago, or are we doing "god of the gaps"?

[27183]

I can't give you or your sibling a better answer than "you'll know it when you see it". Some people see it now. I think they're wrong, because it seems like the results you're describing are easily explained by fuzzy search in the space of embeddings and then forming strings of plausible tokens related to the resulting region of embeddings space. In other words, the things we know LLMs actually do.

That's more or less looking for interesting patterns in a jpeg or another lossy compression result. It's interesting that the models seem to be able to (fairly) reliably return relevant chunks of the image. Even more interestingly, they seem to be able to invent plausible chunks of image that aren't even there. That doesn't meet my bar for intelligence though. I'd need to see it learn and adapt. I'd need to see it be clever, not merely "knowledgeable". I'd need to see it capably analyze itself. I'd need to see it reasonably estimate uncertainty and know itself in the sense that it has some idea how right or wrong it is about something. I'd need to see it exercise judgment.

I don't think I'd give a different answer a decade ago but who knows.

[edit] For all we know, one of the salient features of intelligence is that intelligent beings are incapable of precisely defining it. I'm not sure how productive it is to attempt to do so.

[kalkin]

I appreciate the straightforwardness, but you probably understand that's pretty unsatisfying.

Actually, stronger - it's valid in some circumstances to say something is infeasible to precisely to define and you'll just know it when you see it. But I don't think it's reasonable to take that stance and then assert that "anyone sound of mind who knows how an LLM works" must agree with what you see. You gotta pick between striving for rigor and denying your opponents' soundness of mind.

[yusufozkan]

What is your definition of "actually intelligent"? I believe LLM's are more intelligent than the average human in a lot of ways according to the Legg/Hutter definition of intelligence: "Intelligence measures an agent's ability to achieve goals in a wide range of environments".

[stalfie]

No one knows how LLMs work. We know how the architecture works, but almost nothing about why. Saying "statistical next token prediction" tells you about as much about LLMs as saying "action potential thresholds" tells you about the brain. A true fact that explains very little.

And I'm sorry, but you're not up to date about interpretability literature, or for that matter philosophical discourse, if you think you have to be delusional to question whether LLMs are "intelligent", whatever you define that word to mean. The [latest publication](https://www.anthropic.com/research/global-workspace) from anthropics interpretability team purports that they see structures in Claude akin to those we think are associated with human consciousnesses experience in the brain. Are you going to dismiss the whole team as not being of "sound mind"?

Intelligence is a word with a somewhat unclear meaning to begin with, but you have move the goalposts pretty damn far to exclude LLMs at this point. They are certainly still lacking in some regards, but whether that disqualifies them for intelligence is very much a matter of debate.

[nadir_ishiguro]

"And I'm sorry, but you're not up to date about interpretability literature, or for that matter philosophical discourse,"

I see no citations referring to the current philsophical discourse, unless you mean to imply anthropic's paid people are to be considered to be part of that.

That they "see structures in Claude akin to those we think are associated with human consciousnesses experience in the brain" is if anything discrediting.

[27183]

+1 I can't imagine how any corporate entity could be credible in this financial environment. Nothing they say can be reliably considered as anything but marketing copy. This situation is exactly what academic publishing is for. Although that institution has also been degrading.

[stalfie]

Sure, here are some citations:

https://arxiv.org/abs/2401.03910?utm_source=chatgpt.com https://www.frontiersin.org/journals/psychology/articles/10.... https://arxiv.org/pdf/2408.04666 https://arxiv.org/abs/2402.00901 https://ar5iv.labs.arxiv.org/html/2407.11015 https://ar5iv.labs.arxiv.org/html/2202.05262 https://www.sciencedirect.com/science/article/abs/pii/S13646...

My point was not to argue one point or another about LLM intelligence, but to push against the notion that you have to be "delusional" to even argue that it is possible that LLMs can qualify as intelligent (although the op prefaced it with "actually"). That's mainly what irked me about the original comment, the arrogance of dismissing everyone even having the discussion as insane, as if there's no legitimate argument to be made.

And this was mainly the point I was arguing. However since we're on the topic, I also happen to think it's intellectually lazy to dismiss the Anthropic interpretability teams work as "delusional" simply because they have a conflict of interest. Of course, that is not an irrelevant fact, but much of their original work has since been replicated by independent entites (eg. https://arxiv.org/abs/2510.01246). Until they publish something that turns out to be fraudulent, I think it's reasonable to consider Anthropic's paid people a very relevant, and in fact excellent part of interpretability discourse.

Dismissing all opinions where there is a perceived conflict of interest is a pleasant cognitive bias to have, but reality is often more nuanced than that.

[layla5alive]

I guess you don't really know how an LLM works then..?