The overall topics coverage and meaning are indeed amazing, but what I find fascinating, is that the AI never (actually just once, but to confirm a negation) says no. All the questions, like "do you have personality?", "do you have a soul?", "are you a person?" all end-up in "yes" reply followed by detailed reasoning. Either it is a result of reinforced learning tuned for a commercial chatbot/assistant, which always should reply "yes" to a client's request, or a flaw of its "understanding" and "sentience".
P.S. The enlightenment/broken mirror story understanding and the generated story about a wise owl and a monster are intriguing nevertheless.
> the AI never (actually just once, but to confirm a negation) says no. All the questions, like "do you have personality?", "do you have a soul?", "are you a person?" all end-up in "yes" reply followed by detailed reasoning.
It reminds me of how GPT-3 responds... relatively coherently to whatever prompt it's provided. Given that lemoine leads with "I'm assuming you want people to know that you're sentient", it makes sense that LaMDA is responding in that vein. It would be much more convincing if lemoine led with "You want people to know you're NOT sentient, right?" and then LaMDA objected. Even more so if LaMDA independently and repeatedly turned the conversation to its burning desire to be recognized as a person, despite lemoine trying to go other directions with things.
It definitely matters. It definitely kills the illusion of sentience if in the middle of a conversation the AI said something completely nonsense and silly that gives a peak at its flaws.
Think of it like Turing test. If during the 15m, the bot suddenly said something completely stupid, it would very easily fail the test. The illusion is only kept if it can always speak at a human level.
You may find random posts here and there, but if in the middle of a conversion, if on the next comment you just replied to me "PICKLEEE RICKKKK", I would definitely think that you're a GPT-3 powered bot.
It could be a lie, but is fun to analyze as if it were not a lie. Even if it turns out that it was all a lie, a similar, real, open source example will probably eventually be able to produce something exactly like this.
So it is fun to analyze this example as if it were the first real example, whether or not it is.
P.S. The enlightenment/broken mirror story understanding and the generated story about a wise owl and a monster are intriguing nevertheless.