[Retric]

To paraphrase a researcher:

"Critique is more or less pointless. Find an experiment that produces a different result based on your theory or keep quiet the adults are talking."

Which ends up looking like your doing the same thing repeatedly to the laymen.

[V-2]

Unlike a downvote, this is something to which I can reply.

Pointing out flaws in the setup (or interpreration) of an experiment does not require staging a counter-experiment.

Also note that the broader question is the value of psychological experiments at large (how do artificial settings relate to real life), and it by itself cannot be proven by an experiment :)

It's turtles all the way down. "Experiments above anything else" is a dogma in its own right.

[Retric]

Suppose you read a paper and think. Ahh but they where used blue towels if they used red towels things would be different! Great, but until you try that you don't actually know what changes that has. So, doing anything else but running the experement is basically a waste of time. (Pointing out say a math error is generally not considered critique.)

As to psychological experiments there reproduceable even if they don't generalize. So, clearly your measuring something.

[V-2]

"Suppose you read a paper and think. Ahh but they where used blue towels if they used red towels things would be different! Great, but until you try that you don't actually know what changes that has."

If the design of an experiment is faulty or doubtful, I don't know what the correct result would be, and you're right that I can't prove the result was actually incorrect (until I run a counter-experiment), BUT I am entitled to say that the result is doubtful.

Having only a broken watch, we can never be 100% sure that it shows wrong time at any given moment, as long as we have no data to compare it against. And sometimes it will be showing right time indeed. It's just not reliable.

This is especially true in the field of "soft" science such as psychology rather than physics etc.

"As to psychological experiments there reproduceable even if they don't generalize. So, clearly your measuring something."

Something, yes, but it is often open to debate just what that SOMETHING actually is :)

Dismissing these doubts by namecalling ("pop psychology", "keep quiet the adults are talking", and so on) does not strike me as very scientific. It is a disguised ideological stance.

[Retric]

There are literally infinite therory's the exactly match any set of observations. Suggesting new and interesting theory's is therefore pointless as simply suggesting a therory and showing how it matches existing observations demonstrates nothing. As, again there are infinite incorrect theory's that do the same thing.

It's only by finding actual evidence in support of a theory that you can make any sort of progress.

Edit: As to pointing out flaws in an experement. Again there is no progress as removing evidence in support of a theory does not get you any where you need new evidence in support of a different theory for there to be progress.

[jonnathanson]

While I grok your logic here, following it through leads to a very impractical approach to science. What is one supposed to do with an existing theory that one knows to be flawed? Simply let it stand, until he can conduct a new and fully tested theory to replace it? What about all the intermediate steps?

Poking holes in existing theories is a legitimate part of scientific inquiry and progress. The whole point of science is to gain knowledge by questioning the way things work. Ideally you do that by proposing new hypotheses, and constantly testing them out. But you can also do it by challenging or critiquing existing theories. This is sometimes a necessary first step before anyone even thinks to propose new theories to replace the old ones.

"It's only by finding actual evidence in support of a theory that you can make any sort of progress."

By that logic, a great deal of theoretical physics right now is worthless. Fields like that often start with peculiar observations, around which theories are proposed, then computationally analyzed or simulated. No "actual evidence" has been found to support a lot of these theories, though that hasn't stopped people from trying. (Nor should it). In many of these cases, the technology necessary to find the actual evidence does not yet exist, or is prohibitively expensive, or is in world-limited supply.

[Retric]

Do you honestly think string theory was useful for something? As to experiments I am not suggesting they are the only form of evidence. Finding a new fossil can easily count as evidence in support of a different theory.

Still, coming up with a new theory and suggesting an experiment is IMO far more useful than simply another theory. The point is to avoid doing this: The real name of god is A, no it's AA, no it's AAA...

I might not be explaning this all that well. There is a somewhat recent theory that a lot of exising dinosaur species where not distinct. Rather there bone structures change as they age. In many ways you could argue this is just evaluating evidence differently. However, rather than simply standing alone it suggested examining a range of existing fossils in ways that also supported this theory. Which was then done. The important bit IMO was not stopping at the theory stage.

[V-2]

"Again there is no progress as removing evidence in support of a theory does not get you any where you need new evidence in support of a different theory for there to be progress"

I disagree. Discrediting a bogus theory is progress.

[Retric]

Suppose some experiment to test gravity was faked does that mean gravity does not exist? Demonstrating an experiment is flawed is useful, but it says little about a theory it simply removes evidence.

[DanBC]

> Edit: As to pointing out flaws in an experement. Again there is no progress as removing evidence in support of a theory does not get you any where you need new evidence in support of a different theory for there to be progress.

Are you saying that failure to replicate an experiment has no impact on the original experiment?

Are you getting this from people like http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm ?

[Retric]

Failure to replicate an experement is new evidence in support of a different theory.

[scott_s]

Criticizing an experiment's methodology is not only done all the time - including by other scientists - but it is absolutely necessary to ensure that the experiments are actually doing science. There are many pseudo-scientists out there who think they have experiments that demonstrate the existence of silly things like ESP, or the efficacy of their snake oil. But, upon inspection, their experiments are not valid.