[gwern]

One thing I'm not following is how the side/order bias is being handled. OP measures a IMO very large bias towards left-hand treats, but it is unclear how that is handled (ahem)? Skimming https://github.com/adamwespiser/best-dog-treat/blame/main/an... doesn't help me understand if it is being modeled as a covariate to adjust for the bias or if it was dealt with by construction (eg. by always offering pairs twice, swapping hands), or what?

Incidentally OP if you want to make it more adaptive, you can just fit the B-T model each time, and grab a posterior sample of what the best pair is, and test that, which turns out to be Thompson sampling. I did this for fun with blind taste-testing of mineral waters: https://gwern.net/water

[markerz]

I suspect it’s by construction, looks like they’re offering every permutation of treat choice twice.

Below, I filtered for A vs E, the top two choices. Notice how they switch left and right hand each time:

A/E :: A

E/A :: A

A/E :: E

E/A :: E

A/E :: E

[wespiser_2018]

Author here, yes, that's correct.

The initial set-up was to do 3 comparisons of the 5 treats (30 trials), alternate between right and left hand, then write a quick python script to randomize the order.

A bit more than halfway through the experiment, I ran the model and realized that A/D/E were the only contenders left, so I removed the B/C trials and added more A/D/E trials.

[RataNova]

I liked that the post surfaced the bias instead of quietly ignoring it

[nosioptar]

Some dogs have a strong preference on right/left. I've known at least six dogs that will only shake hands with their left paw.

[kokanee]

Yes, and also the presence of a window fan on that side is a big deal because the selection is based on scent. In my experience dogs appear to "like" a scent based on how strong it is, not necessarily how it is "flavored" (for examples, dogs can behave as if they love the smell of poop or skunk spray, but it's actually the powerful sensory information that they're reacting to, not a culinary desire). It seems possible to me that the dog is choosing whichever scent is more powerful.

[wespiser_2018]

If I repeat this experiment, I'll be a lot more careful about that, or also alternate which direction I'm standing in.

I didn't realize R/L preference was biased until I did the data analysis, and during the trials he was picking from both hands often enough that I perceived it as roughly 50% depending on the treats!

[RataNova]

Yep, I have seen dogs that clearly have "a side" for certain habits, even if it is not obvious whether it is paw preference, training, comfort...

[wespiser_2018]

That's awesome! Thank you!