[pmarreck]

> I see no reason why AI would have any difficulty defining its own reward functions

The first problem is epistemological... If you think that creative decisions are made by complying with a "reward function," you are entirely missing something. Most values are fundamentally based on irrationality. I've literally spent an entire life doing things that everyone else told me was wrong and being interested in things that almost no one else saw the value in, but which ended up being "correct" (for me, at least, and also leading to tangible success). I have no reason to believe that any of my decisions were rational, functional, or acted according to a "reward function"... and I'm a programmer! So I COMPLETELY understand the appeal of the explanatory power of "reward functions." And yet, I can assure you that this is a piss-poor explanation for many creative decisions that literally no one else understands but the person making it, but which then bears fruit despite all reason to the contrary. Some might call this "intuition"

[lgas]

I think perhaps you're just misunderstanding some of the terms you're attempting to use. Those things that everyone else told you were wrong and that no one else saw the value in... your reward function rewards pursuing those. And in that context your decisions were rational and functional.

[pmarreck]

I am not misunderstanding anything. I'm 51 and have been programming since I was 10 in 1982- Rest assured that I know what a "function" is, and I know what "optimizing for a local minima/maxima" is from my machine learning coursework. You can't just say there's a "reward function" without defining it. It's otherwise a completely hypothetical assumption, and assumptions are beliefs, and beliefs are useless from the perspective of rationality. There is otherwise nothing rational about some of the things I felt I needed to do, and yet a very disproportionate percentage of them seemed correct in hindsight.

What YOU have to realize is that you (like many others in the past) can only seem to understand the explanation for something in terms of only what is already understood. And that there is nothing "magical" or "special" about our current understanding (unless you believe there's nothing new to discover, which is preposterous hubris).