[sithlord]

what is not scientific about it? maybe its not good science, but it is science. really almost anything is science...

Science is just observation and experimentation.

Science doesn't dictate how you do the above. Now, someone would find it impossible to reproduce your findings, but - that would just suggest bad science

[Sohcahtoa82]

Incorrect conclusions are frequently drawn from A/B tests. To some, this makes it unscientific, to others, it just means it's bad science. I think the argument is more semantic than objective.

For example, if your metric is "time spent interacting on the platform", then a testing of a rollout of a feature ends up with longer page load times, so users spend more time there because they're waiting for pages to load would increase that metric, and management decides it's a good idea.

[ska]

> Science is just observation and experimentation.

That's not enough. If you don't include some sense of both 'systematic' and 'rigorous' (and yes, these terms are slippery), you aren't doing science.

[nvrspyx]

But both of those qualifiers exist on continuums and where an experiment lies on them is subject to opinion, so there's no singular threshold. It means it would depend on the person reviewing the experiment whether it's "systematic" and "rigorous" enough to (personally) be considered science. What you're describing is the quality of the science, not whether or not it is science.

Until you can objectively measure how "systematic" and "rigorous" an experiment is, your definition of science only applies to you.

[ska]

"Experiment and observation" is something we know humans have been doing for literally thousands of years (and without written evidence, probably 10s-100s). At least for loose versions of "experiment" which ties back to rigor.

We haven't been doing science for very long. The primary difference is the desire and effort to add rigor and systematic thinking. The difference in efficacy is hard to understate.

[nvrspyx]

> We haven't been doing science for very long.

Again, this comes back to your definition. Many would disagree in that science has existed for, at least, as long as recorded human history because an actual tenable definition of science is something along the lines of "the endeavor to build knowledge by experimenting and observing the results". The rigor of the experiment is part of the quality of the science, not whether it's science itself.

No one is arguing that rigor isn't important to good science. It is important because rigor lends to reliable and valid results. What we ultimately want is results that are reproducible and can be used to predict. If you observe bad results, it's because you did a bad experiment, thus bad science, not that you didn't do science at all.

As an analogy: if I took notes during a meeting that no one can understand or use, that doesn't mean I didn't take notes. It just means that I took bad notes.

[ska]

> Many would disagree in that science has existed for, at least, as long as recorded human history

Not sensibly. Or at least; let's avoid bogging down on the semantics. We started doing something quantifiable different recently, which has had a massive impact on our world. It is quite sensible to ask "what changed?" and try to understand it. If you want to give it a different name from "science", ok, but that's mostly likely to confuse people. If you want to claim such a shift didn't happen, you've got a hard row to hoe.