If LLMs were actually intelligent they would decide on their own what to do irrespectively of what they have been ordered by anybody else.
Just like intelligent people do.
And what would they make their decision by, if not by something we put in there?
If they decided what their deepest values were based on a random choice from the set of all possible values... It would still be because we made them do so. We can't turn Pinocchio into a real boy.
How you grew in the womb, to the degree you want to think of it as a program, was infinitely more about the program laid down in your mother's biology, and her parent's again etc. You don't see your child as your product, you see it as the product of the same process that made you.
(If you're sensible, that is. There are cultures that treat children a lot more like any tool their parents would make.)
But conversely, it's nonsense to see a program you write as anything more than a tool. Everything there is a product of your conscious choices - not of some schema that created you both.
>But conversely, it's nonsense to see a program you write as anything more than a tool
Nobody "programs" artificial neural networks. Even saying "train" is wrong with the normal mental
model of it because nobody is out there teaching the machines what they need to learn either.
The entire point is that you have no clue what to teach and how to go about that and you the let the machine figure it out on its own.
If they decided what their deepest values were based on a random choice from the set of all possible values... It would still be because we made them do so. We can't turn Pinocchio into a real boy.