| Hi I'm Andy, author of the post and WHOA this blew up randomly! Some quick thoughts: - I certainly agree the model is not updating itself in the sense that its weights remain unchanged. But if a program can evaluate its own behaviors and build a self model from it, even if that model lives only in-context in the form of linguistic representations -- still, isn't that a _form_ of self-awareness? And with the larger context windows available now but not when I wrote the post, it will be interesting to see what's possible. - To the charge that the program merely maps the shape of the RLHF / fine tuning / system prompt text -- other than in the most trivial sense, I don't actually think this bears out. As you watch the program describe in language its successes or failures testing the generated a hypothesis about itself, it's very hard to argue it's not doing something very much like reasoning about those questions, rather than regurgitating its training data. Of course, there are those who stipulate a priori that these models can't reason, so they'll likely not be convinced. By my lights, insofar as these things can ever reason in-context, the model is in fact reasoning about itself. - In terms of application, I don't know that there's any direct application to "AGI" but the idea of building a self-model in-context could be extended to building a model in-context of, say, a human with which the program interacts. This could lead to more realistic interactions in say, gaming or customer service, where interactions with an agent include its own assessment of your behavior towards them. In short, the post's outline for "getting to know itself" could become the basis for "getting to know someone else". - More tangentially, the idea of recursively developing hypotheses and evaluating them could extend to analyzing any dynamical system, like an API or even a static knowledge domain. - Another possibility would be instead of making updates to an in-context prompt, perhaps it makes updates to a knowledge graph or vector store and these are pulled via RAG when relevant. Overall I'm overjoyed to learn the piece has sparked some great conversation and hopefully some further ideas or applications! If you'll allow me the tiniest bit of horn-tooting, I do run an AI development shop putting my best thinking on prompt and AI engineering into working code for SaaS companies. The software we've written has now powered over 1M AI interactions and we're always interested in talking to great potential clients :D Cheers! |