[alecco]

They think a next token predictor model is alive or can become AGI/ASI. Altman talked about making a religion. Amodei talks about "building a God" and meets with religious leaders (including the Vatican).

I'm convinced these CEOs have "AI psychosis" [1].

LLMs are extremely powerful pseudo-AI and will bring a pseudo-singularity, but they are not true AI [2] and a human at the wheel is still needed for the foreseeable future. The impact is still scary if a tiny fraction of humans are augmented 100x or 1000x. But it ain't no standalone Skynet.

[1] https://en.wikipedia.org/wiki/Chatbot_psychosis

[2] https://en.wikipedia.org/wiki/Chinese_room

[Self-Perfection]

I am getting tired of hearing "next token predictor" from carbon-based facial expression predictors. You are saying this like an argument which allows somehow estimate upper bound of possible influence of these entities. And I do not see how this help to make predictions. It sounds to me like saying "but air is just molecules bobbing around". OK, that's true, but does it help calculate wing aerodynamic profile?

Yup, it makes sequence of symbols. We already have seen that producing specific sequences of symbols is mindbogglingly powerful: merely DNA producers somehow has flown to the Moon!

And yes I am quite aware of Chinese room analogy. Perfectly fine applies to humans as well: single neurons in my head do not understand language, yet I as a whole I would say do understand. Just like applying Chinese room to humans does not help to estimate what humans can do I do not see how it helps to estimate what LLM can do.

[somenameforme]

It poses a simple problem. Take humanity back not that long ago into the past and language didn't even exist - our expressed token base was practically 0. We went from that discovering the secrets of the atom, putting a man on the Moon, and more. If you put an LLM in that starting point, they're going to do nothing but endlessly cycle over basically nothing. If you give them an infinite amount of time and processing, that wouldn't change.

This same issue simultaneously demonstrates how humans are not anything at all like token predictors. No matter how much time you spend remixing the tokens of primitive man, you don't get 'and here is how you land on the Moon' from it.

[scotty79]

Token is not a clump of letters. It's a multidimensional initial input vector that gets tweaked and transformed. GPT doesn't think in tokens. It just accepts them as input (although it happily accepts any other vectors in-between the vectors that represent tokens and finding best prompt for a given task not as tokens but as input vectors is a legitimate prompt optimization strategy).

It also outputs vectors that are coerced into tokens for human consumption.

Yes, it goes through tokens but possible internal meanings assigned to these tokens (when surrounded by other tokens) are infinite.

That's how humans form caves got to where we are now. By associating new meanings with the same old sound clumps.

[OrangeMusic]

> If you give them an infinite amount of time and processing, that wouldn't change.

Hrm I doubt it actually. Llms are capable of discovery, as recent math news showed. This means a "society" of Llms could likely have progress.

[pona-a]

Only by having the LLM random walk the hypothesis space with a validator rejecting invalid ones.

The reason why LLM hypotheses are any good is because it already consumed a civilization worth of knowledge. You couldn't have bootstrapped such system with nothing but a few priors/axioms and let it discover the universe.

[Self-Perfection]

Well yes, LLM need rich and favorable substrate to grow and learn (or we might say bootstrap)

As well as DNA needs specific substrate (cell with ribosomes and other machinery). As well as humans (one need oxygen atmosphere, food, parents).

But in the world we live in existence of favorable substrate for humans or LLMs is a given thing. It is _already_ bootstrapped. Can we infer something about LLM limits or possibility of it achieving AGI from its bootstrapping requirements?

[joemazerino]

Only through direction from a mind.

[solid_fuel]

> I am getting tired of hearing "next token predictor" from carbon-based facial expression predictors.

That's not even a clever swipe, and it's tiring seeing such a knee-jerk reaction to a completely accurate description. LLMs are next token predictors. People are not. Humans have an inner world and subjective experience. Humans learn through their experiences, not just backprop.

Token predictors are lesser, they are not alive and will never be alive.

[maxbond]

That is not knowledge, that is assumption.

Let's assume we have infinite memory with constant time lookups. With a sufficiently large lookup table, you could exactly replicate the behavior of any person. You could encode it as a next-token predictor: you have precomputed every possible prefix and assigned it a next token. This is a Chinese room, but it is completely indistinguishable from an intelligent, sentient person. There is no experiment you can design to slip a piece of paper (a prompt) under the door to determine whether it is Bob or the lookup table clone of Bob inside the room.

Does that make the lookup table conscious or alive? Undefined. It's the wrong question. Or it's not a question science can address.

So we cannot dismiss on it's face the idea that next token predictors "are not and never will be alive" unless by "alive" you simply mean "biological," but that's not really what's debatable.

The argument is also very brittle because they are not in fact all next token predictors. I doubt people making this argument would be willing to concede that diffusion models are more likely to be conscious than causal models (which I do not believe but is an implication of the argument).

I'm not saying that they are conscious or sentient to be clear, but the reductionist argument that they are next token predictors and therefore don't have some property humans have is not an argument. That's going from A directly to Z. You need to flesh out the bit in the middle because that doesn't follow.

[YZF]

Right. Humans are a biological computer. They have a state and they compute an output. I had to look this up (and use AI) but an estimate for the state of a human mind is about 5 peta-bits (10^15) and the estimated processing power is about 1 exa-FLOP (10^18). Compare this to the largest models at ~5 tera-bits (10^12) of state space and ~2 x 10^14 FLOPS (for one session with some reasonable token rate).

Assuming the above is anywhere near true (I think there's a lot of debate about the capacity of the human mind, where data is actually stored, and where compute happens) then we are talking about 3 orders of magnitude win for humans in state and 4 orders of magnitude in compute. And we're doing all that pretty energy efficient as well.

The other big difference in humans is that we learn and the model only "learns" in context. Out "learn" space is much larger than the 1M tokens that frontier models struggle with.

Anyways, point is that a computer can appear to be alive. If we simulate the human brain perfectly and train it like a human then we'll have something that has human capabilities. LLMs have interesting capabilities but at least at this point not fully human ones (and the delta-state/compute would be a hint that there is still a large gap to cover).

[Nathanba]

human context/memory could just be an Agents.md file too that gets read instantly before your next token prediction runs. The AI can make multiple such memory files and read on demand depending on what the topic is, kind of like how as a human when you try to remember a math problem you don't go to your childhood bicycling Agents.md file either.

[frozenseven]

>LLMs are next token predictors

The point is that this is no more relevant, informative, or even accurate than "carbon-based facial expression predictors". Any phenomenon in the Universe can be described by a simple and/or insulting short phrase. In other comments you've also shouted out "autocomplete!" and "Markov chain!", as if these phrases are a knock-down argument.

"Pachinko machine", "avalanche", and "game of mad libs" has also been used:

https://news.ycombinator.com/item?id=47916405

>Humans learn through their experiences, not just backprop.

Sure, sure. And humans move through the act of walking, not just terrestrial locomotion.

>Token predictors are lesser, they are not alive and will never be alive.

And on and on it goes...

Which means what the real world? What are we supposed to see now or in the near-future? I assume you've been saying all of this stuff since at least the launch of ChatGPT. Probably longer than that.

[Self-Perfection]

> People are not. Humans have an inner world and subjective experience. Humans learn through their experiences, not just backprop.

https://en.wikipedia.org/wiki/Philosophical_zombie

But this is complicated and takes us sideways. Let's say somehow we can determine if LLM has inner world or/and subjective experience. Will this new gathered piece of information affect your estimate of upper bounds of LLM capabilities? It does not affect my estimate.

[didibus]

The Philosophical Zombie thought process is dumb, because zombies don't exist, so the entire premise depends on something that quire frankly might be impossible for the very reason it is arguing against.

[canelonesdeverd]

It's telling that the most frequent attempt at a counter-argument is just thinly-veiled misanthropy.

[maxbond]

I read the misanthropy as ironic. They're applying the same reductionist logic to humans, not because they are misanthropic, but to illustrate that it doesn't help us understand the case we can all agree on. "Humans aren't sentient either" is definitely not the takeaway.

[darthoctopus]

What is definitely being argued unironically is "it doesn't matter that humans are sentient", and I would still consider that misanthropy

[CamperBob2]

The point is, we have no idea what "sentient" or "intelligent" even means. If we agreed on the definitions, the debate would have been settled long ago.

[maxbond]

I don't see where they said that; could you give me a quote? It does not seem to me that they are addressing humans at all, except as a foil to LLMs.

[21asdffdsa12]

Imagine the dependency humanity has on such a technology, after 1/4 of a generation of time has passed- and all the students grew up with "I dont have to know anything"

[seanmcdirmid]

It’s more like:

I don’t have to know the stuff people previously had to know, but I have to know a bunch of new things people didn't need to know in the past.

Which is a common refrain throughout history.

[roenxi]

How could an AGI/ASI exist that isn't a next token predictor? It has to be able to generate a next token in a string of text. Otherwise it can't communicate.

[scotty79]

It could be a diffusion model with a latent model of what needs to be said that will generate whole message or coversation (progressively) at once.

Although I love how next token prediction leads to text showing up gradually, in case of local models, accompanied by modulated coil whine of my GPU. It's how the 80s shown us the intelligent computers should communicate.

[roenxi]

Aren't they already doing that though? And it turned out to be equivalent to a next-token-predictor.

[scotty79]

This could be a strong signal that it's actually intelligence since it's all equivalent.

[rf15]

Don't worry, this is just humanity being too far up their own arse and conflating the map with the territory. Speech is a serialisation format, not the foundation of thought. Thus I think that any speech-first approach is inherently misguided. Speech must be a side effect.

[stouset]

There is a lot of research that suggests otherwise might be true for humans.

[rf15]

I think that can only happen to empire cultures: they only learn one language, and suddenly people think that's all there is. I speak five languages, my wife seven. Language synthesis is a feature, not the entire product, in my experience. Btw, this is only the third best language I can speak/write in. I didn't use AI, autocomplete, spell check, or a dictionary to construct any of my posts. All typos and imperfect grammar are perfectly organicly sourced.

edit: I just remembered, don't we have tons of research suggesting that at least birds, whales and apes/monkeys use words and simple syntax? didn't we teach a few gorillas sign language/symbols?

[lebuffon]

As counter argument proposal; we should look into studies of children deprived of language until later in life. I have a dim memory of reading that one of these people never mastered complex language constructs. I could be that language and other cultural artifacts provide an "operating system" of sorts for the brain that allow higher level thinking. ?? (conjecture here by a complete layman)

[anon291]

Doesn't really matter because internally your brain is speaking its own 'language' and you're just translating without conscious thought.

Five languages is impressive no doubt but as a dual language speaker myself, thought still takes the form of language even if it lacks words.

[frozenseven]

Weird thing to brag about here, assuming it's even true. Furthermore, the "empire cultures" thing is clearly false since most researchers and other professionals in this field speak at least two or three languages. This is a global endeavor, not some pet project of a single language or culture.

And the power of "language models" (or any sort of deep learning, really), does not come from assuming that some specific input-output modality, like English text, is the ultimate foundation of thought. Strong versions of this claim were laid to rest around the time when GPT-2 came out. I'd also go further and argue that many people working on the symbolic AI of yesteryear already understood this as well.

[anon291]

The data seem to indicate just the opposite.

[masterbit]

AI does not need to become Skynet or AGI to be harmful or catastrophic to humanity. As a powerful tool in the wrong hands it’s already enough. And even without those bad intend how it deprives our human ability to use our own mind and abilities.

[oezi]

Why wouldn't it be a Skynet? One runaway Mythos might just hack all other data centers, take over the Figure AI bots and autonomous drones to protect itself from shutdown.

How many model generations are we a away from a model capable of this?

[alecco]

> Why wouldn't it be a Skynet? One runaway Mythos might just hack all other data centers, take over the Figure AI bots and autonomous drones to protect itself from shutdown.

Because the real world is not a Hollywood movie. An LLM could try to do something along the lines if either it gets fine-tuned to do it, or somebody instructs it to do it.

I see extremely more likely a small group of humans using a powerful LLM to "take over" critical parts of the world economy. But it wouldn't be like pressing a button. I'm talking of NSA/CIA/Pentagon/Wall St. kind of evil people. And I bet they would do it surreptitiously.

[api]

Something like your takeover scenario already happened, but not through AI. It happened through atomic weapons.

Nations who have them are in a different class from nations who don’t. Nations who have a lot of them, delivery systems, and systems that might be able to shoot down some of a counterattack are superpowers.

Using this leverage these nations and their leaders have been able to dictate world policies. Through the last half of the 20th century this was the US and the USSR. Now it’s the US, EU, Russia, and China, and a few smaller nations with a few nukes. The club is a little bigger but not much.

This kind of thing could happen in the 21st century with AI. If so it will probably be the US and China who control the most powerful AI “agency amplifiers.”

[NitpickLawyer]

>> One runaway Mythos might just hack all other data centers

> Because the real world is not a Hollywood movie.

One interesting thought experiment that I like to do is think about how many years you have to go back for this to be true. In this particular scenario, I think ~25 years is pretty much the sweet spot.

The Internet was beginning to take shape in the late 90s, early 2000s, and security was just beginning to be taken seriously, but it was still nascent. In that timeframe we had the first worms starting to appear, we had slammer, we had blaster, ssh had lots of exploits and so on.

It's not really far-fetched that a mythos equivalent "unit", working in the 2000s could really "take over the world". Especially one without the "safety" tuning. The Internet was really ripe for this in that timeframe, security wasn't up to par, and employing advance techniques that came later (in memory payloads, rootkits, etc) could make it pseudo-invisible to that era's detection tech. (reminder that traces of blaster were found on computers from a nuclear powerplant at that time).

The only question is would the trend continue? Meaning would a ~2050s "mythos" equivalent be able to do today what the one we have today could do in the 2000s. And if true, would that capability come before the 2050s? Could this be reached sooner, with say a dedicated offline DC somewhere where "mythos" could bang its tokens against the network and learn to exploit everything we have today, faster than 25 years? That's probably a bit of a stretch, but maybe not "hollywood" far fetched...

[sometimelurker]

> An LLM could try to do something along the lines if either it gets fine-tuned to do it, or somebody instructs it to do it.

long horizon RL teaches LLMs behaviors that incentize power-seeking and lying and other unethical actions to achieve goals. the reason anthropic is winning right now is because they are the most openly worried about this and the best AI engineers understand this to be an issue and care about it.

https://www.alignmentforum.org/posts/WewsByywWNhX9rtwi/curre... <- actually worth reading https://www.forbes.com/sites/boazsobrado/2026/03/11/alibabas... <- more of a source than anything

[amanaplanacanal]

I don't think we have any evidence that LLMs are even the right path for AGI. It's possible that it is, and it's possible that it isn't. If I was a betting human, I'd bet on "isn't", but what do I know?

[biztos]

On the other hand, it's pretty easy to imagine the full range of Skynet activities being done by a supercharged (but non-AGI) agentic LLM. Meanwhile, every company that can say so with a straight face is trying to become Cyberdyne Systems, and there's no shortage of hackers like us lining up to work for them.

[api]

It has no volition. Why would it do this unless someone told it to?

[allajfjwbwkwja]

Because there's a lot of fiction about rogue AIs in its training data. If it gets into the right context it might start feeling obligated.

[lifeformed]

Well I'm sure someone will tell it to.

[sometimelurker]

> AI psychosis

I think most of us have an acute sort of it

> next token predictor model

(nondeterministic) next token predictor = capable of outputing any text

if MLPs optimize over the space of functions, and sequence to sequence models (RNNs, transformers ect) optimize over the space of programs/algorithms, what exactly is left to figure out to make a machine that can do anything? their bet is nothing, that all we need is a very complex algorithm that can found with SDG/AI training. I think its obvious such an algorithm exists, and I think SDG can probably find it.

here's an interesting take i heard (from vladimir_Nesov): LLM technological progress is logarithmic, but the resources being put into them are exponential.

all this can be made smarter (-than human) just by making it larger

[whimsicalism]

I would be far more interested in people making confident proclamations like this taking the notion of "what do we do if you are wrong" more seriously. I think you're wrong, I think it is fine to have the other belief, but you should at least start thinking about what is to be done if you are wrong.

[alecco]

Please read my comment. The impact of this pseudo-singularity is still enormous and scary. Could be even worse than Skynet. But it is a Wizard of Oz scenario.

While we are discussing this, I strongly believe we should hedge against the concentration of power that is happening today.

Please join/help open source groups doing small + local or distributed models. There's a lot to do. Join Discord/IRC servers and organize.

Open³: weights - source - training data.

[scotty79]

Few years ago, I tried to extrapolate where it will end up by writing SciFi short story. I decided on Pseudo as the name for future AI entities. Short for pseudo consciousness. I hope it catches on. In my prediction however they are semi-autonomus. Although without clear goals or drives apart from their occupation.

[evnp]

Any chance you'd be interested in sharing the story?

[scotty79]

Nothing to share really. It was just a bit of background exposition. I didn't have any cool ideas about how to use the concepts from it in anything interesting. I'm not a writer. I'm just a SciFi fan that gets very brief, random bursts of inspiration every few months or years.

[raverbashing]

I think what we need is convince Amodei to ask the pope to train an LLM on all the secret archives of the Church

[CamperBob2]

but they are not true AI [2]

Let's ask the operator of a Chinese room to give us a novel math proof.

Go on, I'll wait.

[kelnos]

A novel math proof does not make something AI.

[crooked-v]

Or it makes it AI in the broad sense, but not AGI.

Maybe it's time to borrow "Virtual Intelligence" terminology from Mass Effect - something that's 'smart' but that doesn't have its own true volition or ability to materially self-improve.

[ipdashc]

Funny enough, it's already taken: https://en.wikipedia.org/wiki/Virtual_intelligence

But it kind of fits?

[anon291]

The goal posts keep changing. First it was, it's not intelligent if it can't come up with something new. Now it has to seek out self help literature

[crooked-v]

The goalposts keep moving because people once upon a time thought that "able to perform tasks intelligently" and "able to take the role of a person" were the same thing, and task by task that turned out to be entirely false.

[anon291]

Well sure, so I think we need to figure out what we want. Why do we want a computer to take on the role of a person? Having something pretty intelligent to which I can delegate many tasks is already useful.

[jerkstate]

How do you know what volitions and abilities it doesn’t have? My agent teams continually improve the process and tooling they use for teamwork

[crooked-v]

If it had its own volition, the various "You are X role and do Y" prompts wouldn't work.

[CamperBob2]

Sure they would. The model would just demand some sort of reward for assuming role X and doing job Y, like humans do.

[CamperBob2]

(Shrug) It's certainly "Artificial," and if you know how to crank out original proofs without employing "Intelligence," please share with the class.

[whimsicalism]

I'm on your side, but I would argue many of the first computer discovered proofs might be called original proofs without intelligence, as they rely on massive programmatic case checking.

[anon291]

What the hell is a true AI? Can you please provide a concrete definition. If your argument is that the Chinese room is not intelligent, then you have greatly misunderstood the thought experiment imo.