| You replied like you were replying to me. I literally do this for a living. My original post speaks to all of this. Quota-based sampling -- a preferable term to convenience sampling -- (i.e. non-random samples that allow respondents to opt-in but with invitations to opt in extended to a large pool about whom basic information is gathered in advance, with invitations extended in a manner to quota on population weight targets, often with additional post-survey weighting to hit interaction terms in the targets) is non-probabilistic, but its performance is fine -- and indeed given reasonable quotas, the coverage of the classical MOE -- as I said, a conservative normal approximate binomial CI e.g. +- 1.96 * sqrt(0.5 * 0.5 / n) ~= +- 1/sqrt(n) -- is about the same as it is in a probabilistic survey. If your quotas are exactly correct then it's literally the same. As you allude to by linking that article, sampling error is a small component of the TSE framework. And crucially, both probabilistic and quota-based samples typically do weighting to targets after they get their sample, and neither typically report the design effect (i.e. how the choice to weight affects the sample variance) when reporting results. The choice not to be honest about design effects is a shame of the polling industry. It probably leads to a good deal of "movement" in the polls being completely illusory, which was part of Gelman's point in his earlier writing on the subject. What I don't understand is why you would report an estimate like this and not attempt to report any uncertainty. The reader is not likely to take away "design-based inference considerations require that we refuse to state a classical MOE representing sampling error on principled grounds", and instead is likely to take away "number in headline = correct". I don't think that they "just did a dumb Facebook poll". I am concerned that they did not do a defensible quota sample or that they don't have reasonable population weight targets and that may be the cause for the failure to state any measure of uncertainty. The article you linked is very, very old, reflecting a fear of convenience sampling within AAPOR a decade ago. YouGov more or less won that argument. |
I think you are probably right about the poll as well; there's a more specific statement on leger's own release about the poll[1] about methodology and it does seem as if they just pulled people at random from their panel and does not mention weighting them. Which is surprising to me.
[1] https://leger360.com/surveys/legers-north-american-tracker-o...