[red_green_yell]

Are they lazy dismissals? Or a simple observation that LLM scaling is hitting diminishing returns way way short of AGI.

If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?

[roywiggins]

What if "way short of AGI" is still dangerous in ways we can't easily handle?

[overfeed]

When did the pacers have this mass-revelation about dangers of AI? The timing is curious.

Which takes priority: public safety or the free-market + national security? The AI labs hype in now biting them in the ass: why would China and the rest of the world deny their businesses a technology that "is bigger than the invention of electricity" or whatever hyperbole the snake-oil salesmen came up with when they were not breathlessly declaring that the first side to get AGI wins forever.

Even if the pacers are right, it's like greenhouse emissions all over again; each country wants everyone else to cut emissions, but no one wants the economic hit of doing so.

[bpascal]

i think part of the problem, though, is that we've already hit many of the warning signs that AGI is coming, and people have kept moving the goalposts. if you were to tell someone from ten years that there's an ai that has passed the turing test and has made novel mathematical discoveries, i think many would say AGI is already here.

[versteegen]

The old observation that people tend to define AI as whatever computers can't do yet is as true as ever. It's getting a bit absurd, moving from demanding "general intelligence" to replicating human cognitive phenomenology (the experience of cognitive activities). Yet LLMs can already somewhat (confabulation-prone) introspect their own "internal unspoken thoughts" in their residual streams, quite fascinating.

[marcus_holmes]

You're right, in that we would have said that.

I wouldn't frame it as moving the goalposts in a bad way, though. We understand more now than we did then, so we have a better idea of what the actual goal is, and moving the goalposts to frame that goal is more rational than insisting the goalposts stay where they were ten years ago even though the entire pitch has changed since then. I'm straining the analogy, obviously.

[gr_norm]

Moving the goalposts, so to speak, is how science works! We must of course update the things we think based on an improved understanding of how the world works. Only the deeply incurious could consistently demand that one adhere dogmatically to a prior way of understanding the world in the face of changing knowledge. It's important not to treat science as a game to be won (of which 'goalposts' are evocative), but as the collaborative effort that it is.

Note well that I have not said LLMs are not useful, only that no amount of interaction with them has convinced me they remotely approach general intelligence. Their actual function of heuristically regurgitating all information ever known to mankind remains extremely useful in lots of domains.

And by the way, it's not obvious an LLM could fool me (or the average person) over a sufficiently long period of time, exactly because they are heuristic machines. That they lack thought guided by underlying cognitive models seems to always leak out, in the end. I suspect pass rates by LLMs on long-range Turing tests (if there have been any conducted) might drop as people become acclimatized to them.

[bpascal]

what do you mean by 'general intelligence'? i definitely agree that in comparison to humans, they have some definite weaknesses. however, i think they also have definite strengths. and i don't think their current weaknesses imply that much about a potential singularity or ai risk.

[gr_norm]

Sorry, I should've used clearer language. I just mean intelligence in the sense humans possess it. Computers have always been able to exceed limited elements of human intelligence (say, at arithmetic), but the problem is to simultaneously match or exceed all of them. I specifically think that the important parts they currently lack make them a no-go for supposed existential risk.