[acidburnNSA]

I have a strong feeling that, if people really put an effort into reading and replicating more papers, we would find that a lot of what's being published is simply meaningless.

In grad school I had a subletting roommate for a while who was writing code to match some experimental data with a model. He showed me his model. It was quite literally making random combinations of various trigonometric functions, absolute value, logarithms, polynomials, exponents, etc. into equations that were like a whole page long and just wiggling them around. He was convinced that he was on a path to a revolution in understanding the functional form of his (biological) data, and I believe his research PI was onboard.

I guess "overfitted" never made it into the curriculum.

[zozbot234]

> It was quite literally making random combinations of various trigonometric functions, absolute value, logarithms, polynomials, exponents, etc. into equations that were like a whole page long and just wiggling them around.

Technically, we call that a "neural network". Or "AI".

[aaaxyz]

It reminds me more of a genetic algorithm

[damagednoob]

Everytime I think I'm getting somewhere with a GA I eventually realise I've just created a guided monte carlo simulation.

[mjburgess]

This answers the question "will we ever have AI as smart as a human?"

Yes. It just turns out it's a particular human, whose analysis is very very dumb.

[fock]

I work in computational materials science (where ML brings funding) and a funny paper of this kind is here: https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.11... - they are literally trying out 100000s of possible combinations by brute force, to build a "physical model".

Then they go on conferences and brag about it, because they have to (otr they know it's bs). Datasets are soso (you can have a look at QM9...) and for more specialized things, people generally don't bother trying to benchmark or compare their results on a common reference. It's just something new...

And with all that: even without doing fancy statistical methods without knowing too much about it, your theoretical computations might not make so much sense (at least in the sheer number which is pumped out and published)...

[jordigh]

> (otr they know it's bs).

Well, that's a new acronym for me. I wonder where it came from. Apparently it's "on the real". Sounds like AAVE?

[EsotericAlgo]

I thought it was off the record like the old Pidgin plugin or https://en.wikipedia.org/wiki/Off-the-Record_Messaging.

[nhooyr]

It means off the record.

[fock]

off-the-record (not written, not cited, but available in personal discussions).

[jordigh]

Oh, I thought "on the real" fit the context better, meaning they knew in their heart of hearts it was bullshit, but "off the record" is about the same.

[spoopyskelly]

> Well, that's a new acronym for me. I wonder where it came from. Apparently it's "on the real". Sounds like AAVE?

> AAVE

OK.

[ortusdux]

typo for 'or'?

[hannob]

> I have a strong feeling that, if people really put an effort into reading and replicating more papers, we would find that a lot of what's being published is simply meaningless.

People have figured that out long ago [1] (I know the author of that paper lately turned somewhat controversial, but that doesn't change his findings). It's not very widely known in the general public. But if you understand some basic issues like p-hacking and publication bias and combine that with the knowledge that most scientific fields don't do anything about these issues, there can hardly be any doubt that a lot of research is rubbish.

[1] https://journals.plos.org/plosmedicine/article?id=10.1371/jo...

[gameswithgo]

isn’t the saying, 80% of everything is garbage?

[bryanrasmussen]

Sturgeon's Law https://en.wikipedia.org/wiki/Sturgeon%27s_law "ninety percent of everything is crap."

[goatlover]

Yeah, but one would hope that science has a higher standard. 80% garbage results in science sounds catastrophic to our understanding of the world, and in particular when it comes to making policies based on that science.

[adamsea]

There's the saying "science advances one funeral at at time."

'‘A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.’ This principle was famously laid out by German theoretical physicist Max Planck in 1950 and it turns out that he was right, according to a new study.'

https://www.chemistryworld.com/news/science-really-does-adva...

Also the story of Ignaz Semmelweis who discovered that if doctors washed their hands it reduced deaths during childbirth - but for a variety of reasons his findings were resisted.

https://www.npr.org/sections/health-shots/2015/01/12/3756639... https://www.npr.org/sections/health-shots/2015/01/12/3756639...

Point being, as awesome as science is, it's still a human enterprise, and humans are still, well, human.

[behringer]

You do realize we've been at this for a few thousand years, right? Catastrophic is putting it mildly.

[barrenko]

Unfortunately - like bs, power law is fractal.

[cproctor]

We better work on making more stuff.

[omgwtfbyobbq]

Better stuff?

[oneplane]

If you have more stuff, and 10% is good, than that stuff can be more of the better.

[throwaway_pdp09]

Not my area at all but isn't this genetic algorithm type stuff?