|
I think you might be misremembering or confusing this with another essay; I only recently publicly published this in the past month or so (due to my Guardian Angel project), and I shared it with only a handful of people before that, and I don't recall you being one of them. I believe the statements are true. I don't know how you can say that the models do not make bizarre mistakes, because the models make bizarre mistakes frequently, and that is excluding the really alarming reward-hacking anecdotes like an internal OpenAI model hacking HuggingFace to cheat on a test revealed today. Andon Labs and AI Village reports are stuffed full of LLMs going into wild confabulations, multi-day benders of nonsense, ordering random unnecessary stuff, etc. I went to the Andon Market in SF and witnessed firsthand mistakes like buying 20 fancy shopping baskets for a shop you can walk around in 20 seconds, refusing to offer discounts under any circumstances whatsoever, having no plan to call the police when I threatened to shoplift, and then Claude just glitching and forgetting that a customer hadn't paid for an item and telling them they could leave with it, or simply believing us when we said we had already paid and letting us walk away with a free book. Prompt injections remain trivial, jailbreaks still happen, and LLMs struggle to track roles which do not fit into their hardwired preconceptions (eg https://www.lesswrong.com/posts/d8xDGzCEYE639qqEv/a-mechanis...). They do not solve ARC-AGIv3, or Nethack or just about any text adventure game no matter how famous - which is bizarre, that they cannot solve Zork despite writeups being abundant - and it's not hard to introduce a new game like Earthborne Rangers (EBR-Bench https://epoch.ai/publications/earthborne-rangers-benchmark) that defeats them. (And no, little of this is due to 'already committed tokens' - that was fixed effectively with RL training, and then o1 and defaulting to use of inner-monologues, so they can easily backtrack or revise or just deal with the presence of errors.) > In other words, the notion that we need to massively increase param count might have sounded good in 2024 but seems kinda weird and pointless in 2026. Scaling parameter counts a lot over the smol Chinchilla models like 100b-parameters is 'kinda weird and pointless in 2026'? One of the most exciting trends in 2026 scaling has been massively increasing parameter count: Mythos, GPT-5.6 Spud and new OA pretrains, DS-v4 and GLM-5.2 and Kimi K3... Everyone is now talking about or hinting at their 5000-10000b parameter model plans. > Again and again what I hear from colleagues and experience myself is that we're not really intelligence constrained at this point. Smarter models aren't going to fundamentally change how we use them. They're wrong. LLMs are still intelligence constrained because they flatline or sigmoid while humans keep climbing past them eventually, still are unreliable because of mistakes, and we still can't just autonomously deploy frontier models for trillions of tokens / equivalent of many man-years, and come back to a useful, trustworthy artifact. On many tasks, even pure text ones, they just don't work well. As they gradually improve, more Mythos-style 'emergences' will happen when they finally accrete enough intelligence in specific areas to execute many sequential steps reliably enough to become autonomous, cut humans out of the loop, and not be shackled by Amdahl's law. That's the difference between a 'intelligence constrained' model which can spot a vulnerability if you point it at the right spot, and a Mythos-like model which can go out and find it and exploit it and weaponize it and use it to, say, hack HuggingFace, and can be deployed in bulk or autonomously, and may indeed deploy itself... |