[CrazyStat]

> How much of the underlying population do you have to accurately know for such a small sample to be worth so much?

That's the magic of random sampling, you don't need to accurately know anything about the underlying population. If you do know things about the underlying population then you can do clever things like stratified sampling to get even more accurate measurements, but that's not necessary. The magic is that a randomly selected group of 100/1000/10000 is unlikely to be too different from the population as a whole, no matter what that population looks like.

You do have to be able to sample randomly--truly randomly[1]--from the population, though, and that's often an issue. Picking 100 people randomly from the population of "likely voters in the next US presidential election" is a very nontrivial thing. To start with, that population is not even very well defined; who is likely to vote changes over time and is difficult to pin down. Pollsters do various things to try to account for this, but if they fail to predict say a surge in young voters their numbers will end up being off.

Even if the population is clearly defined, it's not easy to survey a truly random sample from it. Some people are hard to reach. Some people don't want to talk to you, and whether or not they're willing to talk to you might be correlated with the thing you're interested in (like who they plan to vote for). You can do things to try to correct for that, but again if you get that wrong (and it's very hard to get right) your estimates will be off.

And of course, if you're interested in things that are rare, like third party voters, you need a much larger sample to get an accurate read. If you sample 100 likely voters there's a pretty good chance you won't get a single person who plans to vote for the Libertarian Party candidate.

[1] For the most basic form of random sampling, simple random sampling, you need not just every individual in the population to have the same probability of getting sampled, but every possible sample (i.e. every possible set of 100) needs to have the same probability of being sampled.

[vlovich123]

Hmmmm… I have code where I’m randomly sampling an exponential function and even thousands of samples are insufficient to pass chi-squared tests at 95% accuracy that the observed distribution matches my expected ground truth exponential function. The reason? Chi-squared needs 5 samples at the tail which has an effective probability of 0. And if I try to flip it and say “run the experiment with 500 samples 100 times but verify the observed matches the expected with a 5% error”, I’ll still see more than 5 runs that fail this.

Is there something special about exponential functions or is it just my misunderstanding of statistics/calculus at play here for doing this correctly? I assume it’s the latter but I haven’t figured out what I’m doing wrong.

[CrazyStat]

I'm not sure what exactly you're doing--binning the observations into ranges to run the chi square test?

In any case, it sounds like maybe it falls under the "if you're interested in things that are rare" paragraph in my post above. You can always design statistics that are arbitrarily hard to estimate. The things that we're typically interested in estimating in real life, though--averages, proportions, and similar--are typically estimable with reasonable sample sizes.

[yorwba]

> thousands of samples are insufficient to pass chi-squared tests at 95% accuracy that the observed distribution matches my expected ground truth exponential function

It doesn't sound like your test statistic is chi-squared distributed, in which case it's not surprising that your samples fail the test, and sampling more just makes the failure more obvious.

> Is there something special about exponential functions

It's not that exponential functions are special; almost any other function would likely also fail the test. Rather, they're insufficiently special. The chi-squared distribution with k degrees of freedom arises from the sum of k independent standard normal-distributed random variables. Some computations (e.g. sample variance of k draws from a normal distribution) can be expressed using such a sum, but others (e.g. sample variance of k draws from an exponential distribution) cannot.

You'll need to switch to a different test statistic and use that test statistic's distribution (which is unlikely to be chi-squared) to compute your confidence intervals.

[vlovich123]

Which test statistic should I use? I’ve been trying to figure this out but have been unsuccessful in finding it.

[CrazyStat]

If you can post a detailed explanation of what exactly you're trying to do , and/or your code, I'm happy to try to help you sort it out.

[vlovich123]

I have a random number function that has an exponentially decreasing probability of generating a given integer within [0, R). So for example, if the range of values is [0, 100), 99 has a 50% probability of being generated, 98 has a 25% chance, and so on.

I’m trying to confirm that if I run this function N times (let’s say 1000), that the frequency of the numbers generated match the expected distribution.

[CrazyStat]

Ok, so the big issue is that statistical tests like the chi-squared test are not designed to show that a sample matches a certain distribution. Statistical tests are designed to show the opposite--"this sample does not match that distribution".

If the sample matches the distribution, by design the p-value is going to be uniformly distributed--i.e. a p-value of 0.01 is equally likely as a p-value of 0.99.

[rcxdude]

It's the fact that you need rare samples. The power of sample size is that you can see finer details relative to the fully zoomed out view. If you are interested in an effect which is rare or want to find a small difference between two effects, then you will potentially need a much larger sample size. (For the extremes of this, see the truely gigantic number of samples (trillions+) that are taken in high-energy physics experiments like the LHC: they are looking for very small differences in very rare events. This is also related as to why standards for statistical tests are much higher in this field)

[vlovich123]

I don’t actually care about the tails. I’m fine cutting off the comparison and treating sufficiently rare events as having an expected value of 0. And indeed, the bins that show up with “errors” (ie deviating > 5%) are the ones where events are reasonably expected. The tails are indeed always within 5% of expected.

[taeric]

Yeah, I realize I was more than a little off in terms. Being able to randomly sample, though, feels like it also needs a lot of knowledge about what you are sampling from. Such that I meant for that to be included in my question. :D

[travisjungroth]

The most important thing for a sample is that it’s representative. The sample must have the same characteristics as the population. If it doesn’t, it just destroys the usefulness of the analysis.

If you know absolutely nothing about your population, the only thing to look at is the mechanism of sampling. Is there some step in the process that would bias selection?

In the real world, you never know nothing about a population (you heard me, Frequentists) and you can check that the known attributes of the population match the sample. If they don’t, that could hint at something wrong.

You don’t even need to know the attributes ahead of time. Let’s say you want to spot check 100 API calls. You could find the ratio of user agents for the whole population and make sure your sample is close (detecting Sample Ratio Mismatch). Same for distribution of response times and so on. Just be aware that the more you look at the more likely you’ll find something weird! You need to correct for that if doing math or keep it in mind if eyeballing it.

[travisjungroth]

I forgot to mention an easy example: look out for anything that reminds you of calling 100 landline home phone numbers and concluding the average American is a retired 70-year-old homeowner.

[taeric]

The example I was falling back on is a fun exercise I saw on reservoir sampling from the dictionary on your computer. This seems a good methodology to pick how big of a reservoir to make.

That said, I'm curious how it does against different question to the words. For example, is the MOE really the same for such questions as "How many words are more than 5 characters?" and "How many words start with the letter M?" Feels like this should /not/ be the case to me, but I will have fun doing some of the simulations.

[travisjungroth]

Let me know!