[thraway180306]

The whole point of Bousquet study was comparability, so it had to be the lowest common denominator, especially in a field that defines SotA differently year to year in each wave of research. Your objection wouldn't embarrass anyone.

[gwern]

As I said, that is not what anyone reading this article would take away. Anyone reading an article littered with misleading statements like "and some of the gains may not exist at all" (repeated twice!) or "Eye-catching advances in some AI fields are not real" or "phantom progress" is not going to conclude, "actually, systems do perform much better than years ago and are increasingly powerful and relevant to the real world, it's just the performance gains do not come from the claimed sources". A reader is going to take away the conclusion "NNs are just as dumb and useless as they were 5 or 10 years ago in many fields, lol AI hype" - which is the opposite of reality.