Before I get to the meat of the post, I want to remind you that our
\nDonorsChoose drive is ending in just a couple of days! A small number of readers have made extremely generous contributions, which
\nis very gratifying. (One person has even taken me up on my offer
\nof letting donors choose topics.) But the number of contributions has been very small. Please, follow the link in my sidebar, go to DonorsChoose, and make a donation. Even a few dollars can make a
\nbig difference. And remember – if you donate one hundred dollars or more, email me a math topic that you’d like me to write about, and I’ll
\nwrite you a blog article on that topic.<\/p>\n
This post repeats a bunch of stuff that I mentioned in one of my basics<\/a> posts last year on the margin of error<\/a>. But given some of the awful rubbish I’ve heard in coverage of the coming election, I thought it was worth discussing a bit.<\/p>\n As the election nears, it seems like every other minute, we <\/p>\n What happens is that they look at polls, and they talk about the results and what they mean. But they, like almost everyone, use the margin of error as if it means something very different than what it really does.<\/p>\n What you hear<\/em> is that, for example, Barak Obama is leading Florida by 5 points, but the margin of error is +\/- 4%, so it’s really not a significant lead. What the journalists seem to think it means The thing is, that’s not what the margin of error means. The margin Polls – and much of statistics in general – are based on the idea of sampling<\/em>. Given a large, relatively uniform population, you can There’s a big problem there: how can you be sure that your sample is representative? The answer is, you can’t! The only way to know<\/em> for certain that a sample is representative is to measure the entire population, and compare the results of doing that to the sample. But once you’ve measured the entire population, what’s the point of looking at a sample?<\/p>\n Fortunately, we can<\/em> assess how likely it is that our sample is a good representation of the population. That’s what the margin of error does – it measures the likelihood of the sample being representative of the population. It’s computed by combining a So when you hear a pollster talking about a “poll of likely voters showing that Obama is ahead by 8 points with a margin of error of +\/-4%”, the big thing you should do is realize what they’re measuring. In that case, the population isn’t “the set of people who are going to vote next tuesday” (even though that’s what the journalists try to make you think); the population is “the set of people who the poll believes are likely to vote next tuesday”. So the margin of error The population being sampled by polls is likely to be quite different from the actual population of voters for a number of reasons, Before I get to the meat of the post, I want to remind you that our DonorsChoose drive is ending in just a couple of days! A small number of readers have made extremely generous contributions, which is very gratifying. (One person has even taken me up on my offer of letting donors choose topics.) […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[61],"tags":[],"class_list":["post-697","post","type-post","status-publish","format-standard","hentry","category-statistics"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4lzZS-bf","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/697","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/comments?post=697"}],"version-history":[{"count":0,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/697\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/media?parent=697"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/categories?post=697"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/tags?post=697"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nhear predictions of the outcome of the election, based on polling. The
\nthing is, pretty much every<\/em> one of those reports is
\nutter rubbish.<\/p>\n
\nis that the margin of error is a total measure of the accuracy of the polls – that the poll result is<\/em> within the margin of error of the “true” result that the poll measures. So, by that interpretation,
\nthe poll is predicting an outcome of 52\/48, and the margin of error means that the range of actual voter preferences ranges between
\n48\/52 and 56\/44.<\/p>\n
\nof error is a statistical measure of the probabilistic size of errors caused by unintentional sampling errors.<\/p>\n
\nget an amazingly accurate measure of that population by looking at a small subset of it, called a representative sample<\/em>. A sample
\nis a randomly selected group that is intended to be a microcosm of the
\nentire population. In an ideal representative sample, the
\nsample must have the same distribution of differences as the population as a whole. <\/p>\n
\nbunch of factors, the primary ones most commonly being the size of the population and the size of the sample. Given those, we can assess how
\ncertain you can be of your measure being pretty close to accurate. Typically, we describe that certainty by stating how large an interval
\nyou need to define on either side of the measured statistic to be
\n95% certain that the “actual” value is within that interval. The size of that interval is the margin of error.<\/p>\n
\nis a measure of how well their poll matches the population of people
\nwho they believe are likely to vote – which is quite a different thing
\nfrom the population of people who actually do<\/em> vote. In fact, it actually does slightly less, even, than that: it measures how much
\nsampling error is contained in their poll due to unintentionally
\nselecting a non-representative sample. That’s not really saying very much in an election poll.<\/p>\n
\nand this difference produces measurement errors that almost certainly significantly outweigh the unintentional sampling errors measured by the margin of error. For example:<\/p>\n\n
\npollsters use when they select people for the sample. For an extreme
\nexample, some polls (like, I think, Zogby) try to get an equal number of
\npeople who self-identify as republicans and democrats. But in most
\nstates, the number of party members in the two major parties are not
\nequal. They are, in fact, often pretty dramatically uneven. A less
\ndramatic but still significant one is that many polls do their polling
\nthrough phone calls, and only call land-lines. Many younger people no
\nlonger have land-lines; the exclusion of cell-phone numbers therefore
\nexcludes some portion of the population from the sample. These kinds
\nof sample bias produce a significant mismatch between the population
\nof real voters, and the population being sampled.<\/dd>\n
\nthat the real population is unknown<\/em>. No one is sure who’s going
\nto vote – which means that no one is certain of what the correct
\npopulation to sample is. Pollsters try to identify a sample of people
\nwho are likely<\/em> to vote. But since the population is unknown,
\nthey don’t know if they’re including people in the sample who aren’t in
\nthe actual population of voters, and they don’t know if they’re
\nexcluding people from their samples who are<\/em> going to vote. In
\nthis election, this is likely to be a significant effect, because huge
\nnumbers of people registered to vote for the first time, but no one
\nknows<\/em> how many of those newly registered voters are likely to show
\nup and vote. Once again, there’s a problem related to the fact that the
\npopulation that they’re sampling isn’t the same as the population that
\nthe poll is trying to measure – so that error factor is outside the
\nmargin of error.<\/dd>\n
\nquestion is phrased. “Who are you going to vote for?” will
\nlikely generate different results from “Are you going to vote for Obama or McCain?”, which will likely generate different results from
\n“Do you plan to vote for McCain or Obama?”, which will generate different results from “Do you plan to vote for a Democrat or a Republican in the presidential election?”. This is a well-known problem,
\nbut it still has a significant effect.<\/dd>\n
\nquestions honestly. A frequently discussed version of this is called
\nthe Bradley effect. The Bradley effect is a phenomenon where people
\nare reluctant to admit to being racist. So when a pollster asks them
\nif they’re going to vote for a black man, they’ll say “yes”, but when
\nit actually comes to voting, they’ll vote for the white guy. I’ve heard
\nsome people speculate on a reverse Bradley effect this year in some southern states, where people are reluctant to admit that they’re going to vote for a black man, so they lie and say they’re voting McCain. But the truth of the matter is, we don’t know if the people answering the
\npolls are answering honestly. If they’re not, that skews the poll results, and once again, it’s not covered by the margin of error.<\/dd>\n<\/dl>\n","protected":false},"excerpt":{"rendered":"