[loup-vaillant]

> There are people who argue that having cyclic preferences is not only rational, but even sometimes the only rational representation of evaluations.

Wouldn't be the first time otherwise serious people are defending nonsense. Noted nonetheless.

> It is perfectly possible to make rational decisions with cyclic preferences.

It is perfectly possible to make rational decision while being insane. Just, not all decisions will be rational. Cyclic preferences are not insane with respect to all decision, but they do mean the decision system as a whole is not flawless.

While the absence of cyclic preferences is of course not sufficient for perfect rationality, it's obviously required.

> By the way, Expected Utility falls prey to Dutch Books. There is a money pump against every risk-averse or risk-seeking agent.

Well, if you're not evaluating risks correctly to begin with, of course you're gonna get ripped off (I'm not saying that's a good thing). Being either risk seeking or risk averse looks like a flaw too, though perhaps less severe than cyclic preferences.

> There is a vast array of literature on qualitative decision making for which this assumption does not hold.

Wait a minute, this one is only talking about epistemology. Jaynes does not mention utility functions at all, and for all I know those may still be allowed to be discontinuous. (That would be perhaps a bit surprising, but I have yet to have an opinion on that particular point.)

Discontinuous probabilities, that would be more surprising. Though I reckon this continuity business is the weak link here. It would be nice if we didn't have to assume it.

> even in probabilistic settings it completely controversial how to deal with conflicting evidence and how and when to revise beliefs in the face of evidence that directly conflicts with your existing beliefs.

There are lots of reason why a piece of data might not change one's mind, even if that piece of data seems to contradict their beliefs directly. For instance that piece of evidence might have been cherry piked from a mass of otherwise normal data.

Not even acknowledging the piece of data might be a good approximation in some cases, but in general it seems quite foolish. You don't just ignore a piece of evidence, you explain why it doesn't change your mind. (I believe Jaynes gives examples of beliefs diverging when exposed to the same piece of evidence.)

> But you seem to be hell-bent on taking Jayne's book as some sort of bible, which is weird.

Call it confirmation bias, but when I read that book, I already subscribed to probability theory as the correct way to think. I had for a long time. The intuition of probabilities being degrees of beliefs, I had for as long as I can remember.

Then this book comes up, and provide justifications for my intuitions that were even stronger than I anticipated. It's like suspecting there's a giant bearded man behind that cloud, and then actually see it. And take a photo, and show it to your friends. Perhaps not foolproof, but pretty damn close.

---

Now we still have a problem. Jayne's Robot cannot exist. I mean, that would be something like AIXI, that's not tractable. Probability theory is not tractable (we wouldn't have Monte Carlo methods if it were, we'd just compute the probabilities directly). Any inference engine that runs in the real world (like humans), has to be imperfect. We have to take shortcuts, and from them, flawed reasoning will arise.

There's also the problem that thinking has a cost. It takes time and energy, and with those, utility. So not only a real engine will have flaws, it also needs to evaluate whether minimising those flaws is worth the trouble (and that evaluation also costs some thinking).

To take a concrete example, the first Alpha Go program lost one of its games in part because it failed to take more time in a particularly hard to evaluate game state. It was obvious to top human players that this particular move required more thought than usual, but the machine wasn't programmed that way.

As certain as I am that probability theory is the correct ideal to attain, I also have to admit that it is just that: an impossible ideal. How to instantiate that ideal into a good enough working implementation, I have no freaking clue.

[jonathanstrange]

Well, I agree mostly with you, except that I'm not a probabilist. There is an extensive discussion about what and what not Dutch books show, see Alan Hájek's work on that, which is really worth reading.

> Being either risk seeking or risk averse looks like a flaw too, though perhaps less severe than cyclic preferences.

Yes, I don't want to deny that this view is appealing. However, even if you use probabilistic representations of degrees of belief, you need to deal with ignorance and conflicting evidence in one way or another. Convex sets of probabilities can be shown to be able to represent many of the alternative approaches I've mentioned. There is also this "subjective logic" by Jøsang that is surprisingly nice despite it's silly name. Check it out, maybe the only quirk I have with it is that he mostly seems to re-brand many prior ideas, but the framework is interesting.

> Not even acknowledging the piece of data might be a good approximation in some cases, but in general it seems quite foolish. You don't just ignore a piece of evidence, you explain why it doesn't change your mind.

I agree with you, but at the same time we know from qualitative belief revision theory that there are many, many ways of dealing with conflicting evidence. Okay, we can rule out some of them, e.g. discarding all previous beliefs to learn the new evidence, but among the many less obviously flawed methods a choice needs to be made. The probabilistic setting doesn't help too much in that area, it actually makes it harder to see what's going on. As I've said, the problem is underdetermined.

> Then this book comes up, and provide justifications for my intuitions that were even stronger than I anticipated.

I'm definitely going to read it! However, I might already be tainted by other books on the subject and philosophical discussions. I really do think a belief representation ought not be closed under negation, i.e., I have strong Dempster-Shafer intuitions, and that some way of distinguishing ignorance from doubt is needed.

> Call it confirmation bias, but when I read that book, I already subscribed to probability theory as the correct way to think. I had for a long time. The intuition of probabilities being degrees of beliefs, I had for as long as I can remember.

Kudos to you for having such strong intuitions. It makes life easier. Maybe I'd be willing to buy into them for probabilities, but that wouldn't help me because of similar problems on the evaluative side on which most of my work focuses. On the evaluative side we have thought experiments like Spectrum Cases (Temkin, Rachels): Suppose A gives you extremely high pleasure for a month, B gives you a little bit less pleasure than A (barely noticeable) for 3 months, C gives you a little bit less pleasure than B (barely noticeable) for 9 months, and so on. Some people (not all) have the intuition that B is better than A, C is better than B, and so forth, until at some point, say Z, they would judge that A is better than Z. These thought experiments come in all varieties, can also be made about well-being and other notions of goodness and can be made as realistic as one wishes. Most people who want to keep "better than" transitive introduce some notion of significance, which is lexicographic decision making in disguise (significant value attributes always outrank insignificant value attributes). But okay, you were talking about probabilities only and already acknowledged the evaluative component could be discontinuous. (To be more precise, in this case the Archimedean axiom fails.) It's just that even if graded belief is purely probabilistic, these kind of preferences will complicate making decisions on the basis of your belief.

I agree about the tractability, too. Since you are a probabilist about graded belief, that already makes your life much easier than mine, though. Couldn't you just say that any heuristics are permissible in certain circumstances as shortcuts that - under these circumstances - are conducive to adequate probability approximations?

I didn't want to insinuate that there is anything wrong with being a probabilist, it's in the end a matter of intuitions, I merely wanted to point out that there are some fairly well-known authors who are not probabilists about graded belief in the narrow sense, e.g. Bouyssou, Fishburn, Vincke, Pirlot in decision making, people like Halpern, Dubois, Prade, Spohn and their scholars in A.I., and of course almost everybody in mathematical psychology such as Luce and Tversky. But as I've said, most of their generalizations can be represented by more complicated probability representations such as sets of probabilities.

Anyway, it was nice chatting with you!

[loup-vaillant]

> Well, I agree mostly with you, except that I'm not a probabilist.

Good enough for me. :-) (Aumann's agreement theorem notwithstanding, I have to recognise the capacity for actual humans to agree is limited.)

> I'm definitely going to read [Jayne's book]!

I have yet to read it all, but the foundations are laid out early. The preface mostly explains where the author is coming from, chapter 1 and 2 do most of the justifications. The rest focuses more on applications of probability theory. My general impression was like:

  Matches my intuitions,
  solid theoretical foundations,
  works in practice...
  ...case closed I guess.

> On the evaluative side we have thought experiments like Spectrum Cases (Temkin, Rachels): Suppose A gives you extremely high pleasure for a month, B gives you a little bit less pleasure than A (barely noticeable) for 3 months, C gives you a little bit less pleasure than B (barely noticeable) for 9 months, and so on.

Hmm, that's a hard one. Depending on the value I attach to pleasure, there should be 3 possibilities: A is best, Z (or whatever is the last iteration) is best, or there's a sweet spot in between. I get the circular preference, and would likely fall prey to it if the circle is hidden to me. But stuff like that is like a big warning sign that most probably requires more thought than a quick intuitive judgement.

> Couldn't you just say that any heuristics are permissible in certain circumstances as shortcuts that - under these circumstances - are conducive to adequate probability approximations?

I could. I didn't because once we start making shortcuts, evaluating the impact on the shortcut on the final assessment is very difficult. Probabilities are non-linear, it's very easy for a seemingly innocuous approximation to translate to snowball into a huge error. But if careful, it often can (and does) work.

It's a bit like floating point approximations. The ideal math on real numbers is correct, floating points only introduce small errors, but some operations (like a division close to zero) can magnify those errors from "approximate" to "utterly wrong". Special care is typically taken to ensure that does not happen.

> Anyway, it was nice chatting with you!

For me as well. I'll keep your references in mind, and thank you for your patience.