{"id":856,"date":"2010-04-29T21:46:23","date_gmt":"2010-04-29T21:46:23","guid":{"rendered":"http:\/\/scientopia.org\/blogs\/goodmath\/2010\/04\/29\/iterative-hockey-stick-analysis-gimme-a-break\/"},"modified":"2010-04-29T21:46:23","modified_gmt":"2010-04-29T21:46:23","slug":"iterative-hockey-stick-analysis-gimme-a-break","status":"publish","type":"post","link":"http:\/\/www.goodmath.org\/blog\/2010\/04\/29\/iterative-hockey-stick-analysis-gimme-a-break\/","title":{"rendered":"Iterative Hockey Stick Analysis? Gimme a break!"},"content":{"rendered":"

This past weekend, my friend Orac sent me a link to an interesting piece
\nof bad math. One of Orac’s big interest is vaccination and
\nanti-vaccinationists. The piece is a newsletter<\/a> by a group calling itself the “Sound Choice
\nPharmaceutical Institute” (SCPI), which purports to show a link
\nbetween vaccinations and autism. But instead of the usual anti-vac rubbish about
\nthimerosol, they claim that “residual human DNA contamintants from aborted human fetal cells”
\ncauses autism.<\/p>\n

Among others, Orac already covered the nonsense<\/a>
\nof that from a biological\/medical
\nperspective. What he didn’t do, and why he forwarded this newsletter to me, is because
\nthe basis of their argument is that they discovered key change points in the
\nautism rate that correlate perfectly with the introduction of various vaccines.<\/p>\n

In fact, they claim to have discovered three different inflection points:<\/p>\n

    \n
  1. 1979, the year that the MMR 2 vaccine was approved in the US;<\/li>\n
  2. 1988, the year that a 2nd dose of the MMR 2 was added to the recommended vaccination
    \nschedule; and<\/li>\n
  3. 1995, the year that the chickenpox vaccine was approved in the US.<\/li>\n<\/ol>\n

    They claim to have discovered these inflection points using “iterative hockey stick analysis”.<\/p>\n

    <\/p>\n

    First of all, “hockey stick analysis” isn’t exactly a standard
    \nmathematical term. So we’re on shaky ground right away. They describe
    \nhockey-stick analysis as a kind of “computational line fitting analysis”. But
    \nthey never identify what the actual method is, and there’s no literature on
    \nexactly what “iterative hockey stick analysis” is. So I’m working from a best
    \nguess. Typically, when you try to fit a line to a set of data points,
    \nyou use a technique called linear regression. The most common linear regression method is
    \ncalled the “least squares” method, and their graphs look roughly like least-squares
    \nfitting, so I’m going to assume that that’s what they use. <\/p>\n

    What least squares linear regression do is pretty simple – but it takes a
    \nbit of explanation. Suppose you’ve got a set of data points where you’ve got
    \ngood reason to believe that you’ve got one independent variable, and one
    \ndependent variable. Then you can plot those points on a standard graph, with
    \nthe independent variable on the x axis, and the dependent variable on the y
    \naxis. That gives you a scattering of points. If there really is a linear
    \nrelationship between the dependent and independent variable, and your
    \nmeasurements were all perfect, with no confounding factors, then the points would
    \nfall on the line defined by that linear relationship.<\/p>\n

    But nothing in the real world is ever perfect. Our measurements always
    \nhave some amount of error, and there are always confounding factors. So
    \nthe points never<\/em> fall perfectly along a line. So we want some way of
    \ndefining the best fit<\/em> to a set of data. That is, understanding that there’s
    \nnoise in the data, what’s the line that comes closest to describing a linear relationship.<\/p>\n

    Least squares is one simple way of describing that. The idea is that the
    \nbest fit line is the line where, for each data point, you take the difference
    \nbetween the predicted line and the actual measurement. You square that
    \ndifference, and then you add up all of those squared differences. The line
    \nwhere that sum is smallest<\/em> is the best fit. I’l avoid going into detail about
    \nwhy you square it – if you’re interested, say so in the comments, and maybe I’ll write a basics
    \npost about linear regression.<\/p>\n

    One big catch here is that least-squares linear regression produces a good result
    \nif<\/em> the data really has a linear relationship. If it doesn’t, then least squares
    \nwill produce a lousy fit. There are lots of other curve fitting techniques, which work in
    \ndifferent ways. If you want to treat your data as perfect, you can use different techniques to
    \nprogressively fit the data better and better until you have a polynomial curve which
    \nprecisely includes every datum in your data set. You can start with fitting a line to two points; for
    \nevery two points, there’s a line connecting them. Then for three points, you can fit them precisely
    \nwith a quadratic curve. For four points, you can fit them with a cubic curve. And so on.<\/p>\n

    Similarly, unless your data is perfectly linear, you can always<\/em> improve a fit by
    \npartitioning the data. Just like we can fit a curve to two points from the set; then get closer
    \nby fitting it to three; then closer by fitting it to four, we can fit two lines to a 2 way partition
    \nof the data, and get a closer match; then we can get closer with three lines in a three way partition,
    \nand four lines in a four way partition, and so on, until you have a partition for every pair of adjacent
    \npoints.<\/p>\n

    The key takeaway is that no matter what<\/em> you data looks like, if
    \nit’s not perfectly linear, then you can always<\/em> improve the fit by
    \ncreating a partition.<\/p>\n

    For “hockey stick analysis”, what they’re doing is looking for a good
    \nplace to put a partition. That’s a reasonable thing to try to do, but you need
    \nto be really careful about it – because, as I described above, you can
    \nalways<\/em> find a partition. You need to make sure that you’re actually
    \nfinding a genuine change in the basic relationship between the dependent and
    \nindependent variable, and not just noticing a random correlation.<\/p>\n

    Identifying change points like that is extremely tricky. To identify it,
    \nyou need to do a lot of work. In particular, you need to create a large number
    \nof partitions of the data, in order to show that there is one specific
    \npartition that produces a better result than any of the others. And that’s not
    \nenough: you can’t just select one point that looks good, and see if you get a
    \nbetter match by splitting there. That’s a start: you need to show that the
    \ninflection point that you chose is really the best<\/em> inflection point.
    \nBut you also really need to go bayesian, and figure out an estimate of the chance
    \nof the inflection being an illusion, and show that what the quality of the partition
    \nthat you found is better than what you would expect by chance.<\/p>\n

    Finding a partition point like that is, as you can see, not a simple
    \nthing to do. You need a good supply of data: for small datasets, the
    \nprobability of finding a good partition is quite high. You need to do
    \na lot of careful analysis.<\/p>\n

    In general, trying to find multiple partition points is simply not
    \nfeasible unless you have a really huge quantity of data, and the slope change
    \nis really dramatic. I’m not going to go into the details – but it’s basically
    \njust using more Bayesian analysis. You know that there’s a high probability
    \nthat adding partitions to your data will increase the match quality. You need
    \nto determine, given the expected improvement from partitioning based on the
    \ndistribution of you data, how much better of a fit you’d need to find after
    \npartitioning for it to be reasonably certain that the change wasn’t an
    \nartifact.<\/p>\n

    Just to show that there’s one genuine partition point, you need to show a
    \npretty significant change. (Exactly how much depends on how much data you
    \nhave, what kind of distribution it has, and how well it correlates to the line
    \nmatch.) But you can’t do it for small changes. To show two genuine change points
    \nrequires an extremely robust change at both points, along with showing that
    \nnon-linear matches aren’t better that the multiple slope changes. To show
    \nthree<\/em> inflection points is close to impossible; if the slope is
    \nshifting that often, it’s almost certainly not a linear relationship.<\/p>\n

    To get down to specifics, the data set purportedly analyzed by SCPI
    \nconsists of autism rates measured over 35 years. That’s just thirty
    \nfive<\/em> data points. The chances of being able to reliably identify
    \none<\/em> slope change in a set of 35 data points is slim at best. Two?
    \nridiculous. Three? Beyond ridiculous. There’s just nowhere near<\/em>
    \nenough data to be able to claim that you’ve got three different inflection
    \npoints measured from 35 data points.<\/p>\n

    To make matters worse: the earliest data in their analysis comes from a
    \ndifferent<\/em> source than the latest data. They’ve got some data from the
    \nUS Department of Education (1970->1987), and some data from the California
    \nDepartment of Developmental Services (1973->1997). And those two are measuring
    \ndifferent<\/em> things; the US DOE statistic is based on a count of the number of 19
    \nyear olds who have a diagnosis of autism (so it was data collected in 1989 through 2006);
    \nthe California DDS statistic is based on the autism diagnosis rate for children living in
    \nCalifornia.<\/p>\n

    So – guess where one of their slope changes occurs? Go on, guess.<\/p>\n

    1988.<\/p>\n

    The slope changed in the year when they switched from mixed data to
    \nCalifornia DDS data exclusively. Gosh, you don’t think that that might be a
    \nconfounding factor, do you? And gosh, it’s by far the largest (and therefore
    \nthe most likely to be real) of the three slope changes they claim to
    \nhave identified.<\/p>\n

    For the third slope change, they don’t even show it on the same graph. In
    \nfact, to get it, they needed to use an entirely different<\/em> dataset from
    \neither of the two others. Which is an interesting choice, given that the CA DDS
    \nstatistic that they used for the second slope change, actually appears
    \nto show a decrease<\/em> occurring around 1995. But when they switch datasets,
    \nignoring the one that they were using before, they find a third slope change
    \nin 1995 – right when their other data set shows a decrease<\/em>.<\/p>\n

    So… Let’s summarize the problems here.<\/p>\n

      \n
    1. They’re using an iterative line-matching technique which is, at
      \nbest, questionable.<\/li>\n
    2. They’re applying it to a dataset that is orders of
      \nmagnitude too small to be able to generate a meaningful result for a
      \nsingle<\/em> slope change, but they use it to identify three<\/em>
      \ndifferent slope changes.<\/li>\n
    3. They use mixed datasets that measure different things in different ways,
      \nwithout any sort of meta-analysis to reconcile them.<\/li>\n
    4. One of the supposed changes occurs at the point of changeover in the datasets. <\/li>\n
    5. When one of their datasets shows a decrease<\/em> in the slope, but another
      \nshows an increase, they arbitrarily choose the one that shows an increase.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

      This past weekend, my friend Orac sent me a link to an interesting piece of bad math. One of Orac’s big interest is vaccination and anti-vaccinationists. The piece is a newsletter by a group calling itself the “Sound Choice Pharmaceutical Institute” (SCPI), which purports to show a link between vaccinations and autism. But instead of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[8],"tags":[],"class_list":["post-856","post","type-post","status-publish","format-standard","hentry","category-bad-statistics"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4lzZS-dO","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/856","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/comments?post=856"}],"version-history":[{"count":0,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/856\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/media?parent=856"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/categories?post=856"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/tags?post=856"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}