[Davidzheng]

Re: But they aren't alive, they don't live in the world and have experiences, and they can't create something truly new.

Is it possible for a character in a novel to have novel experiences? Or for you to experience a novel dream? I would argue yes. You can know the rules of the environment and the starting conditions, but with a bit of randomness (or not) you can generate from that novel experiences which were unexpected - so too from the data & distribution that AIs are already trained on they can experience new experiences.

Another source of novelty is from good verifiers/recognition of a class of object which is hard to construct but easy to verify - here the AI can search and from that obtain novel solutions which were unthought of before.

N.B novelty itself is basically trivial - just generate random strings. But both of the above are mechanisms to generate novel samples inside some constraint of "meaningfulness"