[simondw]

Someone's actual preference is stored in their memory. If you give the LLM access to a database that includes "preferences", I'm sure it will be able to retrieve that.

[michaelmrose]

If it's preference is a variable that exists because a human thinks it would make sense to another human it has no connection with reality.