[thorum]

The article’s conclusion doesn’t seem to match the data they collected. They found that most people (52%) do read faster with bionic reading, the effect is generally quite small, but large for some minority of users - at least one user read 293 WPM faster with Bionic Reading!

The authors then average results (good and bad) across all users, resulting in a number close to zero, and conclude Bionic Reading doesn’t work for anyone, even calling it a placebo effect at the end.

The problem is, all brains are not the same. It doesn’t have to work equally well for everyone to be valuable.

[civilized]

People collect all sorts of fascinating data but then analyze it so superficially, throwing very low-hanging fruit away without a thought. The biggest loss is information about individual variation.

Why do we only crunch data down to averages? Differences in group variation can be measured just as rigorously as differences in group averages. Repeatedly testing an unusual individual is perfectly legitimate and scientifically interesting. There is no Law of Science that says you have to execute a boring protocol that rigidly assumes everyone is the same.

There's a classic story about how the Air Force learned that no one is average, and switched from fixed "average" cockpit fittings to adjustable ones, greatly reducing accidents. But it seems like no one has learned from this.

For heaven's sake, let's try to learn about individuals and what works for them!

[tristanho]

The results are what you would expect if you measured two identical fonts, ie random.

You can make the _claim_ in your comment, but there isn't statistical evidence that this effect is more than random, or if the same people were tested again their results wouldn't be opposite. Put another way, it doesn't pass the null hypothesis.

[thorum]

I think the best approach would be a graph showing the distribution of users across different WPM differences, and if possible a second experiment with two non-BR fonts (to see how much random variation is expected). The positive results might all be noise, but I don’t think that should be the default assumption.

[tristanho]

FWIW, the article does include some data on the distribution, as well as address your exact objection:

> Since posting this experiment, I've received a lot of side comments along the lines of, "Well, of course I don't expect Bionic Reading to work for most people, but for [my subpopulation], it really works." If that were the case, we might expect to see disproportionate benefits for those participants who read faster with Bionic Reading than for those who read faster without Bionic Reading. Let's look at how many participants read faster with each font and their average speed gains.

Table 3: Summary speed differences per faster font Count Percent Delta (WPM) Bionic 998 52% 35 Non-Bionic 918 48% 43

> The number of people who read faster with Bionic Reading was slightly greater (52%) than the number of people who read faster without Bionic Reading (48%). That said, those who read faster with Bionic Reading only picked up 35 words per minute on average. In contrast, those who read faster without Bionic Reading picked up 43 words per minute. It does not appear that when Bionic Reading works, it really works.

[thorum]

Yes, but they’re still averaging the results in the “it worked” category. Hypothetically, if it worked very well for 5 users and only a little for 95 users, an average score over all 100 will make it appear to not work for anyone. I’m not going to argue this is definitely the case - just that this analysis doesn’t account for the possibility, and it wouldn’t be unexpected in the area of reading ability, where one would expect individual differences to be important.

[thaumasiotes]

> the effect is generally quite small, but large for some minority of users - at least one user read 293 WPM faster with Bionic Reading!

Well, that's not really something you'd want to admit in your paper, much less advance as an argument. The obvious conclusion is that there's a mistake in your data, not that some users will see an increase in reading speed of 4.9 words per second.

[gnicholas]

Reminds me of The End of Average, [1] a book about human variation. It talks about how the Air Force tried to design a cockpit by averaging various measurements, only to find that their pilots weren't actually average.

Your point is especially apt here, where there is a possibility that the technology could be helpful for a subpopulation, and act somewhat like an assistive technology. It is very tricky to assess the utility of assistive technologies (I work in this field and have asked many experts how they do it) because the average impact isn't the most important thing. It doesn't matter if high-contrast mode is bad for 90% of people, if for the 10% who would actually use it, it's helpful.

One way that the study could have been optimized is if participants had been asked (after taking the test, to avoid priming) if they have any specific reading challenges. That would have helped identify whether there appear to be any subgroups that disproportionately benefit from the approach.

1: https://www.amazon.com/End-Average-Succeed-Values-Sameness/d...

[tpoacher]

I think the idea here isn't that we're treating all brains the same, but that the deviations represent statistical noise/outliers rather than an actual effect fron the bionic intervention.

I agree though, it would have been an interesting follow-up if they asked only people who felt they benefit from it, and then conducted their analysis again on that sample!