If you measure the wrong thing, you get the wrong answer: Down's syndrome in Britain

One of the blogs I read regularly is Ben Goldacre’s “Bad Science”. I recommend
it highly. (Which reminds me that I really need to find some time to update my blogroll!) In saturday’s entry, he discussed a BBC Radio documentary that described how Britain is becoming a much more welcoming place for Down’s syndrome babies.

Ben did a good job of shredding it. But I also wanted to take a stab, focusing on
the mathematical problem that underlies it, because it’s a great example of two very
common errors – first, the familiar confusing correlation and causation, and
second, using incorrect metrics.

There’s one basic fact underlying this discussion: in England and Wales, the
number of live births of Down’s babies has risen by about four percent between 1989 and 2006.

From this, the BBC’s documentary concluded that Great Britain has becoming more welcoming to Down’s babies. How do they conclude this? Well, since pre-natal testing
was first introduced, the number of Down’s baby live births steadily decreased – until this most recent measurement, when the number started to trend upwards again. Based
on this, they assume that it’s clear that more parents are deciding to
have Down’s babies.

Right there, you can see the correlation/causation problem. The number of
Down’s babies being born has increased; the main factor that reduces the number
of live Down’s births is abortion; therefore, the increase must be caused by people
choosing not to have abortions.

What’s wrong with that? To answer, we need to look at what other factors
are involved in this. First and foremost, what’s the absolute number of Down’s babies being conceived? If that rate has increased, then the number of live births could
increase without the rate of abortion changing at all. Unfortunately, we don’t have
any real numbers for that; it’s not (as far as I know) reported publicly.

But we can try to infer it, by considering what causes Down’s syndrome. Down’s
is a genetic disorder where the mother’s egg cell (it’s almost always the mother’s
genetic contribution that causes Down’s) ends up with an extra copy of chromosome 21,
so that when it’s fertilized, the cell has three copies of that chromosome. That causes a large number of developement abnormalities; people with Down’s have distinctive facial features, tend to be severely retarded, have very thick tongues, and a host of other things. But what causes the egg cell to screw up during
division and wind up with an extra chromosome 21? We don’t know – but we know that
it correlates very strongly with the mother’s age. A 25 year old mother has
a risk of about 1 in 1600 of conceiving a Down’s baby. At 35 years old, the risk rises to around one in 350. (The reason why most doctor’s only recommend amniocentesis for
mothers 35 or older is because that 1 in 350 is the crossover point where the likelihood of the baby having Down’s corresponds to what most potiential parents
consider an acceptable level of risk to the fetus from the procedure.)
(Note: in the original post, I screwed up and said Down’s was related to chromosome 23; it’s not 23, it’s 21. Thanks to commenter Alex for pointing that out!)

The population of women having children in Great Britain has changed somewhat since 1989 – it’s decreased slightly; not dramatically, but slightly, because people on average are having fewer children. That should, arguably, decrease the number of live Down’s births, since the overall pool of pregnant women has decreased. But the other factor that needs to be considered is the age of the women getting pregnant. And that has changed quite dramatically. Since 1989, the percentage of
women getting pregnant who are 35 or older have increased by two and a half times – from around 6% to around 15%!

The upshot of this is that the number of Down’s pregnancies has almost certainly increased quite dramatically.

The underlying error here is that the BBC documentary confused the number of live births of Down’s babies with the percentage of Down’s pregnancies that ended with live birth. Their argument is that the increase in the number of
live births has increased, therefore more parents are deciding to have Down’s babies. But they’re using the wrong metric. The absolute number of people deciding to have Down’s babies has increased – but that doesn’t mean that “more people in society” are deciding to have Down’s babies. When you look carefully at the numbers, it actually
appears that there’s been an increase in the percentage of Down’s babies
being aborted: the increase in number of live births is smaller than
the increase in number of Down’s pregnancies.

So in fact, more parents are still choosing to abort Down’s babies – continuing the existing trend; as testing becomes more available and less risky, more
parents are opting to test, and to abort if they find out they’re carrying a Down’s baby. The rate of abortion for Down’s pregnancies has actually increased, not decreased.

This is what happens when you don’t really understand what you’re comparing.
To do a meaningful comparison of a trend, you need to understand what the actual
things being measured are, and make sure that you’re using a metric that
meaningful measures the quantity you want to discuss. The BBC documentary fouled
that up horribly: they looked at some data, chose an invalid measure for it,
and then using that measure, drew a conclusion based on a faulty correlation,
and used that faulty correlation to infer an invalid cause. Bad math on multiple
levels.

0 thoughts on “If you measure the wrong thing, you get the wrong answer: Down's syndrome in Britain

  1. Anonymous 2

    Except, Anonymous, that a survey of member of the Down’s Syndrome Association is similar to surveying the members of a Jaguar owners club. They may have very nice things to say about their cars and why they chose them, but that doesn’t tell you anything about whether overall Jaguar sales are up or down. It’s wonderful that there is support for families with high-need children, but the numbers still indicate that the rate for terminating Downs Syndrome pregnancies is rising faster than the number of births.

    Reply
  2. Pen

    This very point occurred to me as soon as I read the original article. I’m glad somebody’s pointing out. Now, just to round things off, the other point that occurs to me regularly has to do with the social forces that push women into later pregnancies. Let’s see now, how long does it take these days for a woman to pay off her study debts, get her career at least started, get into a financially secure and stable situation, whilst picking up a potential father along the way? I managed it by the time I was 31, but I’m catching up on the later stages of academic qualification even now. I expect (no, make that hope!) to restart my career at a higher level, aged about 45 or 50!

    Reply
  3. Welcome to Illinois

    Actually what the BBC did was to take one set of numbers (from the National Down’s Syndrome Cytogenetic Register indicating an increase in the number of children born with Down’s syndrome from 717 in 1989 to 749 in 2006) and another set of numbers (from a survey of 1,000 Down’s Syndrome Association members about their reasons for continuing with their pregnancy) and assumed a direct relationship between the two.
    The figures on Down’s syndrome diagnosis and termination rates are freely available in this report: http://www.wolfson.qmul.ac.uk/ndscr/reports/NDSCRreport06.pdf which the BBC and other initial news reports overlooked, even though it was linked to by the DSA press release cited in the above comment.
    The stats indicate that termination rates are pretty much unchanged while the percentage of diagnosed cases resulting in a live birth dropped from 69% in 1989 to 40% in 2006.
    I’ve written more about this here: http://welcometoillinois.wordpress.com/2008/11/24/statistics-provide-mixed-account-of-downs-syndrome-in-uk/

    Reply
  4. Anonymous

    Just to be clear: there is no chromosome 23 in humans… It stops at 22 (2 of each), plus X and/or Y = 46 (23 pairs).
    But great post!

    Reply
  5. Anonymous

    For a programmer errors like these just looks like what a good type system prevents, just like we already do with physical quantities.

    Reply

Leave a Reply