[unishark]

There are definitely studies criticizing ML publications similarly. As a kind of statistics (but often without the rigor), screwups make ML methods appear better than they really are. Hence the literature is packed with screwups.

Other CS subfields that get a lot of criticism are "network science" and bioinformatics.

[dehrmann]

I've been playing with some ML lately. I'm sure some of this is because I only have a high level understanding of what I'm doing, but it really feels like throwing things at the problem and see what sticks and is fast. Add features, remove features, try different network architectures, different activation functions, etc. I think the best understood thing might be overfitting, which is oddly reassuring.

[unishark]

> ...throwing things at the problem and see what sticks and is fast.

That generally isn't enough to get published nowadays though, at least in a simple sense (in a broader sense that process might describe all research, of course). To get published requires some deeper demonstration of a new kind of method that not only works, but is superior to all that preceded it in some important way. In other words you show your new method compared to other methods, where yours must be better. Obviously, bad research here can show a supposed advantage by either doing a unfair job applying competing methods, or overfitting their new methods. Or both. As I understand it, the first one is quite common: comparing a poorly-tuned old method to a carefully-tuned new method.

[jack_pp]

Well yes. I remember hearing that there are now meta-networks, networks that optimize other networks by mechanisms you described.

[garden_hermit]

ML has the benefit though of a rapid turnaround time from Academia -> Industry. Things that work/replicate will be immediately used to make money. Things that don't work will be abandoned pretty quickly (at least outside of ML researchers).