[mdbco]

This consensus algorithm looks pretty good, but it seems it could get into trouble in cases where the distribution of outcomes is multimodal. One thing that is mentioned is that users would be reporting on several events simultaneously (let's say k events), but it seems entirely possible that there could be strong consensus among k-1 events, but multimodality in the reported outcomes for the kth event, making it very difficult to see who is wrong and who is correct for that event.

I know that Augur is planning to allow people to report event outcomes as "invalid", and that might clear up some of these cases, but what about an event where the outcome initially appears to be objective, but after-the-fact it is unclear and open to interpretation, resulting in two distinct camps of reported outcomes (and hence multimodality)? Perhaps one solution would be to simply declare events with strongly multimodal properties in the distribution of reported outcomes as invalid, and thus avoid the somewhat arbitrary decision where one mode would be declared the "consensus" by the algorithm, costing reputation to those who reported the other mode as the outcome.

[joeykrug]

In events without a clear outcome (say greater than 65% certainty, this is a tunable parameter) there's an audit system where a different group of reporters is asked to answer the question. The idea is that they'd notice the extreme multimodality of the last group who reported on it, and in self interest of preserving their reputation, report an "invalid" outcome.