[qb45]

> Now just formulate some sensible statistical model and voila - falsified.

I think it was Carl Sagan who said that "soft sciences" are actually "cargo cult sciences".

Out of your ten people, two are astrology haters and respond that everything is false, two are astrology nuts and respond that everything is true, two respond at random because they can't believe you are bothering them with such bullshit, two respond at random because they think you are CIA agent trying to read their mind, two respond mostly true because horoscopes are deliberately vague and they will find confirmation for most of the claims if they try hard enough.

So you get 55% true. With sufficiently large sample size p-value goes down and you claim that astrology is valid.

As if your aunt hasn't been telling you that for your whole life...

[arrrg]

In this case you would not look at how correct people think the horoscopes are, you would look at whether they think their own horoscope is most correct (they don’t know which is which of the ten). The average “subjective correctness” of the horoscopes is not something you would be interested in. You might throw it in somewhere in the paper you are writing, but more to show that ceiling or floor effects are not an issue (i.e. nearly everyone thinks the horoscopes are all either completely correct or completely false and as such you cannot really properly measure – like measuring the distance between New York and Los Angeles with a yard stick or the size of a molecule with a yard stick).

I dislike the outcome this experiment is measuring. I’m not an expert on horoscopes (and wouldn’t want to be one), but to me the primary purported function of horoscopes is to predict the future. As such people might not necessarily be able to say beforehand whether their own horoscope is more correct than that of someone else†. How should they know? But that is easily fixed: Just ask them to rank the horoscopes after the purported predicted time frame …

Altogether this seems like a solid setup to me, though. It would be fun to conduct this, but ultimately pointless. Astrology is just a too low hanging fruit.

(A potential issue might be people’s awareness of stereotypes associated with their astrological sign. I’m not sure how big a role those play of how important and consistent across different astrologers those are – as I said, not an expert – but if stereotypes associated with astrological signs are consistent and generally well known people might be able to identify the correct horoscope just based on the knowledge of those stereotypes, not any predictive power of the horoscopes.)

It was Feynman using the term cargo cult science. And he was not only talking about soft science, he was more generally talking about all of science. And it was not a general condemnation, it was about saying that “cargo cult science” is always a possibility and obviously an issue, but not that it is always an issue. Though I would definitely agree with you in saying that quantitative soft sciences suffer the most from cargo cult thinking.

It’s a paradox, really. “Soft science” is actually really hard. Measuring people and people’s behavior is a nightmare. There are so many variables, so many complex, ever-changing systems. We are at the limits of what we can achieve and our measuring instruments are extremely blunt and imprecise.

Because it’s so hard we are currently not even trying to solve many of the hard problems there. Because we just don’t know how. As such we limit ourselves to simple stuff and the people who are adept at doing the hard stuff (but not as hard as soft science, since that is so hard, no one knows how to properly do it) are doing things that are easier to nail down, with fewer variables – like physics.

And quantitative soft science limits itself by necessity to simple stuff. It simplifies without understanding the underlying complexity. And often slips into cargo cult thinking doing that …

—

† Though I actually think this is the real function of horoscopes: People have to be able to identify with them and think that they fit them when they are reading them. It’s entertainment. That’s why they are full of platitudes and not very specific.

[wolfgke]

> (A potential issue might be people’s awareness of stereotypes associated with their astrological sign. I’m not sure how big a role those play of how important and consistent across different astrologers those are – as I said, not an expert – but if stereotypes associated with astrological signs are consistent and generally well known people might be able to identify the correct horoscope just based on the knowledge of those stereotypes, not any predictive power of the horoscopes.)

Good point.

[qb45]

> Just ask them to rank the horoscopes after the purported predicted time frame …

And that's the experiment I meant. Some will lie to swing the result to their liking, some will respond with bullshit for whatever other reasons. And the rest may easily respond "mostly true" if the author of those horoscopes has enough skill to make them sufficiently truistic and vague, which is what they typically do. Like, "something important will happen to your life in the next month because Jupiter blah blah blah".

My point is: horoscopes are unfalsifiable by design. Or by evolution with mutations, selection and stuff - whatever floats your boat.

[arrrg]

But that’s not the experiment that’s being conducted.

The question you ask is whether people are able to identify their own horoscope, not whether they think the horoscopes are correct, i.e. they might well believe that on a 1 to 10 scale their horoscope is a 7, but if they then also score all the others at 7 (on average) then the horoscope had no predictive power. Basically, all the other horoscopes act as a control. It’s a quite elegant setup.