[haliyat]

There are a ton of different promising AI approaches explored by researchers. When I was at MIT in the early 2010s, when Deep Learning was just taking off, it was seen as one of a suite of new exciting techniques. For example, some of the grad students that taught the AI classes I took were hyped on an approach called Probabilistic Programming, which adds a complete suite of programming concepts (eg. If-statements) to Bayesian networks allowing you to write extremely concise and powerful programs that can learn based on data and handle uncertainty during an execution. Also while I was there Geoff Hinton gave a series of master lectures on the future of AI after Deep Learning and he talked a lot about an approach called Inverse Graphics: basically treating images as one output from a graphics rendering pipeline that includes a scene with geometry and lighting, projection transformations, etc and then trying to learn all the parameters of that pipeline from images so that you produce not just a classifier output but a whole scene description. Both really cool and exciting approaches that build on top of deep neural nets but aren’t bound to them.

One of the negative effects of the huge hype wave (hype tsunami is maybe more appropriate) around LLMs and genAI generally is that it starves these other approaches of resources (as well as discouraging people from exploring other new approaches). This is what LeCunn is responding to. I know some zealots believe that “bigger LLMs” is all we need for AI progress forever, but based on the entire history of the field, a number of technical issues with LLMs, and the nature of progress of LLMs in the last few years I would described this view as blinkered and risky at best. The field often advances fastest from the early years of new approaches rather than massive over-investment in a single approach based on some early promising results. Historically the later approach tends to lead to AI winters.

[haliyat]

Also, FWIW, I saw a bunch of demos of “token sequence learning” that did a lot of the applications that people have been so excited about with LLMs: producing text descriptions of video and images, text summarization with question answering, etc. Those demos were a little janky and limited and obviously only at the academic paper with impressive video demo stage which is a far cry from fast and reliable enough to be useful in production. But they weren’t categorically different from what we’ve seen with transformers and LLMs. This is one of the reasons I’m more skeptical about claims that transformers + more data and compute is all we need for AGI. After a decade plus of not just MASSIVE compute and data scaling but some fairly clever new techniques I would describe progress as incremental rather than transformational beyond those older results. Honestly, people have forgotten this now, but the biggest change that ignite the LLM hype was the UX decision to present interactions with these models in the framework of a conversation with an agent. This is a trick that goes back at least as far as Eliza and it’s effect is mainly in how it primes the user to think about and relate to the tech. That is also an area where more work can be done (conversational interfaces are not the One Solution to all computing). I recommend googling Interactive Machine Learning, which is its own sub-discipline that specifically studies this problem of how to build UX that is native to, and takes best advantage of, ML/AI techniques to produce software that people can use to accomplish real tasks.