{"id":279,"date":"2007-01-18T13:58:25","date_gmt":"2007-01-18T13:58:25","guid":{"rendered":"http:\/\/scientopia.org\/blogs\/goodmath\/2007\/01\/18\/basics-standard-deviation\/"},"modified":"2007-01-18T13:58:25","modified_gmt":"2007-01-18T13:58:25","slug":"basics-standard-deviation","status":"publish","type":"post","link":"http:\/\/www.goodmath.org\/blog\/2007\/01\/18\/basics-standard-deviation\/","title":{"rendered":"Basics: Standard Deviation"},"content":{"rendered":"

When we look at a the data for a population+ often the first thing we do
\nis look at the mean. But even if we know<\/em> that the distribution
\nis perfectly normal, the mean isn’t enough to tell us what we know to understand what the mean is telling us about the population. We also need
\nto know something about how the data is spread out around the mean – that is, how wide<\/em> the bell curve is around the mean.<\/p>\n

There’s a basic measure that tells us that: it’s called the standard deviation<\/em>. The standard deviation describes the spread of the data,
\nand is the basis for how we compute things like the degree of certainty,
\nthe margin of error, etc.<\/p>\n

<\/p>\n

Suppose we have a population of data points, P={p1<\/sub>,…,pn<\/sub>}. We know that the mean is
\nthe sum of the points (pi<\/sub>s) divided by the number of
\npoints |P|. The way to describe the spread is based roughly on the concept
\nof the average<\/em> difference between the points from the mean.<\/p>\n

So what happens if we naively compute the average of the difference between the mean and the the data points? That is, compute the mean difference? That is – if the mean is M, and the average distance is d<\/em>, then can we use the following?<\/p>\n


\n\"mean-distance.mmf.jpg\"<\/p>\n

Unfortunately, that won’t work. If we work it through, what we’d find is that by the definition of mean, that average difference d<\/em> will
\nbe 0. After all, the mean is the point in the center<\/em> of the
\ndistribution – that means that a simple sum of the differences will be zero – the values larger<\/em> than the mean (which will be positive) will be precisely equal to the sum of the values smaller<\/em> that the mean (which will be negative), and so the sum, and therefore the average must be 0.<\/p>\n

How do we get around that? By making all of the distances positive. And how do we do that? Square them. The standard deviation, which is usually written σ is a root mean-square<\/em> measure – which means that it’s the mean (average) of
\nthe square root of the difference between the points and the mean squared. The sum of the squares is also a useful figure, called the variance<\/em>; the variance is just the mean of the squares – that is σ2<\/sup>. The standard deviation written in equational form, where M is the mean, and P is the set of points, is:
\n
\n\"sdev.mmf.jpg\"<\/p>\n

Let’s run through an example. Take the list of salaries from the mean<\/a> article: [ 20, 20, 22, 25, 25, 25, 28, 30, 31, 32, 34, 35, 37, 39, 39, 40, 42, 42, 43, 80, 100, 300, 700, 3000 ]. The sum of these is 4789. There are 24 values. So the mean (rounding off to 2 significant figures) is 4789\/24 = 200. So what’s the standard deviation? <\/p>\n

    \n
  1. First, we’ll compute the sum of the squares of the differences:
    \n(20-200)2<\/sup> + (20-200)2<\/sup> + (22-200)2<\/sup> + (25-200)2<\/sup> + … + (700-200)2<\/sup> + (3000-200)2<\/sup> =
    \n32400+32400+31684+30625+30625+30625+29584+28900+28561+28224+27556+27225+26569+25921+25921+25600+24964+24964+24649+14400+10000+10000+250000+7840000 = 8661397.<\/li>\n
  2. Then we’ll divide by the number of points: 8661397\/24 = 360891. So the variance is roughly 360,000.<\/li>\n
  3. Then take the square root of the variance: the square root of 360,000 = 600.<\/li>\n<\/ol>\n

    So, for our salaries, the mean is $200,000 with a standard deviation of $600,000. That right there should be enough to give us a good sense that there’s something very strange about the distribution of numbers here – because salaries can’t be less than zero, but the standard deviation is three times<\/em> the size of the mean!<\/p>\n

    But what does the standard deviation mean<\/em> precisely? The best way to define it is in probabilistic terms. In a population P with roughly normal distribution, mean M, and standard deviation σ: <\/p>\n

      \n
    • 2\/3s of the values in P will be
      \nwithin the range M +\/- σ.<\/li>\n
    • 95% of the values will be within the range M +\/- 2σ.<\/li>\n
    • 99% of the values will be within the range M +\/- 3σ<\/li>\n<\/ul>\n

      For any<\/em> population P with mean M and standard deviation σ, regardless of whether the distribution is
      \nnormal:<\/p>\n

        \n
      • At least<\/em> 1\/2 of the values in P will be within the range M +\/- 1.4σ.<\/li>\n
      • At least<\/em> 3\/4 of the values in P will be within the range M +\/- 2σ<\/li>\n
      • At least<\/em> 9\/10s of the values in P will be within the range M +\/- 3σ.\n<\/ul>\n

        If you have a population P which is very large, you often want to make
        \nan estimate about the population using a sample<\/em>, where a sample
        \nis a subset P’ ⊂ P of the population. Since the standard deviation of the sample is generally slightly smaller than the standard deviation of the population as a whole, we add a correction factor for sampled populations. In the equation for the standard deviation, instead of dividing by the size of the sample, |P’|, we divide by the size of the sample minus one: |P’|-1. The ideal correction factor is a lot more complicated, but in practice, the “subtract one from the size of the sample” trick is an excellent approximation, and so it’s used nearly universally.<\/p>\n

        Next topic in the basics will be something closely related: confidence intervals and margins of error.<\/p>\n","protected":false},"excerpt":{"rendered":"

        When we look at a the data for a population+ often the first thing we do is look at the mean. But even if we know that the distribution is perfectly normal, the mean isn’t enough to tell us what we know to understand what the mean is telling us about the population. We also […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[74,61],"tags":[],"class_list":["post-279","post","type-post","status-publish","format-standard","hentry","category-basics","category-statistics"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4lzZS-4v","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/279","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/comments?post=279"}],"version-history":[{"count":0,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/279\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/media?parent=279"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/categories?post=279"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/tags?post=279"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}