Hacker News new | ask | show | jobs
by vbs_redlof 3827 days ago
I'm confused with the "goals and "values" terminology, I've never once heard of these terms in this area of research. Aumann's agreement theorem simple: two agents with common priors cannot agree to disagree if their posterior beliefs are common knowledge.

Beliefs are common knowledge if I know that you know that I know .... (and so) on that something is true. This occurs anytime there is trade between two agents, such as in stock markets. If I can see that you can see that this is the price we're trading at, then the traded price is common knowledge.

One of the curious implications of this theorem is that no rational agent in any market would ever agree to trade with another rational agent.

So Aumann's agreement theorem is really a warning against applying game theory to every scenario. Specifically, the common prior assumption limits the usefulness of game theory in real settings. He makes no mention of "goals" or "values", only the assumptions that the priors beliefs of the two agents are the same and their posteriors are common knowledge.

In the context of the article, disagreement means either: 1) one of you is not Bayesian rational or cannot update probabilities properly (bounded rationality) or 2) both of you must have different priors (subjective beliefs).

Both points are likely to be the case in reality, although we dislike the the second point since once we begin to entertain the idea that agents have subjective priors, you can rationalize anything and "anything goes".

1 comments

Needs are what your neural architecture has evolved to optimize (e.g. sustenance, pain avoidance and affiliation). Goals are the preferred states that the brain learns to optimize its needs (via reward). Values are cultural or (fortunately, but also necessarily due to evolution) Schilling point memes which manifest as virtual rewards that mostly enable human cohabit (e.g. you are good if you don't litter the street, i.e. that will increase your chance of future social reward).