[narrator]

Well let's say that someone criticizes a "honor" system person for being violent. Someone interested in the truth would not react violently to that criticism because they would acknowledge that as a contradiction and that doing so would prove their critic correct. Someone only interested in "honor" points would react violently toward the critic because that's how you get "honor" points, by getting revenge for insults. The person interested in the truth can recognize a flaw in their fitness function. The person only interested in "honor" points never questions the rules. Their is no procedure to question the rules in the "honor" points system just as a computer is unable to question its programming.

[yorwba]

> Someone interested in the truth would not react violently to that criticism because they would acknowledge that as a contradiction and that doing so would prove their critic correct.

But wouldn't proving their critic correct give them "truth" points?

Presumably, you actually mean "someone I'd call a reasonable person, who can recognize the truth; but who has other aims beyond making things true". That person's actions will still follow some kind of points system, although it might be slightly more complicated. But "honor" points can be complicated as well, since they need to encode which actions are "honorable".

Ultimately, the only difference is that you agree with one kind of behavior over the other. But what you think is a flaw in someone's utility function, they'll think is a flaw in yours. That doesn't mean all value systems are equally good, just that their relative ranking depends on the person doing the evaluation.

[narrator]

The fundamental aspect of the truth system is that the person remains flexible as to the way to achieve their goal, but not necessarily the goal itself.

The methods of the "honor" points system never changes in response to any empirical feedback. The "honor" points are the way and the end in themselves. They are not subject to backpropagation in the "honor" points system. They were burned into ROM and the "honor" automotaun faithfully executes the program until it dies. The "honor" system does not evolve. The truth system does not know what it will be in several decades, but it will seek truth and incorporate feedback. It is fundamentally incomplete and will remain so. The "honor" points system is complete -- forever.