[bluGill]

> Plus, the idea that we can remove such small, noisy confounding factors is just silly. We need to look for the things that stand out from that noise floor

We have found most of them, and all the easy ones. Today the interesting things are near the noise floor. 3000 years ago atoms were well below the noise floor, now we know a lot about them - most of it seems useless in daily life yet a large part of the things we use daily depend on our knowledge of the atom.

Science needs to keep separating things from the noise floor. Some of them become important once we understand it.

[jerf]

I don't think we have found most of them. I think we make it look like we've found most of them because we keep throwing money at these crap studies.

Bear in mind that my criteria are two-dimensional, and I'll accept either. By all means, go back and establish your 3% effect to a p-value of 0.0001. Or 0.000000001. That makes that 3% much more interesting and useful.

It'll especially be interesting and valuable when you fail to do so.

But we do not, generally, do that. We just keep piling up small effects with small p-values and thinking we're getting somewhere.

Further, if there is a branch of some "science" that we've exhaused so thoroughly that we can't find anything that isn't a 3%/p=0.047 effect anymore... pack it in, we're done here. Move on.

However, part of the reason I so blithely say that is that I suspect if we did in fact raise the standards as I propose here, it would realign incentives such that more sciences would start finding more useful results. I suspect, for instance, that a great deal of the soft sciences probably could find some much more significant results if they studied larger groups of people. Or spent more time creating theories that aren't about whether priming people with some sensitive word makes them 3% more racist for the next twelve minutes, or some other thing that even if true really isn't that interesting or useful as a building block for future work.

[EvgeniyZh]

So 3% is not interesting but the difference between 10^-7 and 10^-8 probability that there is no effect is interesting somehow?

[bluGill]

Meta analysis after enough small studies show the effect exists.

[kazinator]

Individual atoms, or small numbers of them, may be beneath some noise floor, but not combined atoms.

A salt crystal (Lattice of NaCl atoms) is nothing like a pure gold nugget (clump of Au atoms).

That difference is a massive effect.

So to begin with, we have this sort of massive effect which requires an explanation, which is where atoms then come in.

Maybe the right language here is not that we need an effect rather than statistical significance, but that we need a clear, unmistakable phenomenon. There has to be a phenomenon, which is then explained by research. Research cannot be inventing the phenomenon by whiffing at the faint fumes of statistical significance.

[ramblenode]

> We have found most of them, and all the easy ones. Today the interesting things are near the noise floor.

The noise floor is not static. A major theoretical advance spurs an advance in instrumentation, which then supports more science. The hypothesis space is usually much larger than the data space, making the bottleneck theory, not data. The "end of progress" has been lamented again and again since before Galileo, only to be upended by a paradigm shifting theory that paved the way for lots of new science. Many of these theories were developed long after the data and instruments were available, and were produced with relatively simple data: Young's double slit experiment, Mendelian genetics, the photoelectric effect, Brownian motion, most of classical mechanics, quantum teleportation, BOLD MRI, etc.

[exporectomy]

Doesn't it make a difference if it's near the noise floor because it's hard to measure (atoms) or if it's near the noise floor because it's hardly there (masks)? Maybe if these "hardly there" results led to further research that isolated some underlying "very there" phenomena, they would be important, but until that happens, who cares if thinking about money makes you slightly less generous than thinking about flowers? If they're not building on previous research to discover more and more important things, then it doesn't seem like useful progress.