Publicly released polls almost always (and should always) have a margin of sampling error (MOSE) specified in any report of the results. That is the + or – 3 percentage points, or + or – 2 percentage points you generally see at the end of a method statement. (We often talk about a MOSE as if it applies to the whole survey, but note that in reality it applies to each answer in the survey as you see in all the examples.)
Basically, the margin of sampling error is the price you pay for not talking to everyone in your population group. The MOSE describes the range that the answer likely falls between if we had talked to everyone instead of just a sample. For example, for a telephone sample of 1000 randomly selected adults nationwide, the finding from the poll will be within plus or minus 3 percentage points of the answer we would have gotten had we talked to all 210 million adults. So if one of the findings of the poll was that 58% approved of the job their Governor was doing, we would know that the true value would lie somewhere between 55% and 61% if we had talk to the whole population in the state.
To be technically correct, we really only have some degree of confidence around the MOSE we calculate for probability based samples. Generally, pollsters calculate the MOSE using a 95%confidence level. That is, in 95 times out of a 100, we expect the answer we get from the survey is reflective of the true answer within the MOSE.
What’s so special about 1000 respondents?
MOSE is largely driven by the size of the sample. The larger the sample, the smaller the MOSE, while the smaller the sample the larger the MOSE. Wonder why most polls interview 1000 people? Just look at the chart below. Notice that as the sample size increases the MOSE falls, quite dramatically between small sample sizes of say 100 and larger sample sizes of say 1000. But once we get to 1000, we don’t see much change in the MOSE. In fact, note that between doing 1000 interviews and doing 2500 interviews the MOSE falls by only 1 percentage point. Again, notice that moving between 2500 and 5000 interviews the MOSE falls by only another percentage point. And since one of the main drivers of the cost of a poll is how many interviews you do, there isn’t much “bang for your buck” when you double your sample to these larger sample sizes.
