{"id":536,"date":"2007-10-25T12:15:00","date_gmt":"2007-10-25T12:15:00","guid":{"rendered":"http:\/\/scientopia.org\/blogs\/goodmath\/2007\/10\/25\/tag-teaming-with-orac-bad-bad-breast-cancer-math-in-jpands\/"},"modified":"2007-10-25T12:15:00","modified_gmt":"2007-10-25T12:15:00","slug":"tag-teaming-with-orac-bad-bad-breast-cancer-math-in-jpands","status":"publish","type":"post","link":"http:\/\/www.goodmath.org\/blog\/2007\/10\/25\/tag-teaming-with-orac-bad-bad-breast-cancer-math-in-jpands\/","title":{"rendered":"Tag-Teaming with Orac: Bad, Bad Breast Cancer Math in JPANDS"},"content":{"rendered":"

My friend, fellow ScienceBlogger, and BlogFather Orac asked me to take a look at a paper<\/a> that purportedly shows that abortion is a
\ncausative risk factor for breast cancer, which he posted about
\nthis morning<\/a>. When the person who motivated me to start what’s turned out to be a shockingly
\nsuccessful blog asks for something, how could I possibly say no? Especially when it’s such a great example
\nof the misuse of mathematics for political purposes?<\/p>\n

<\/p>\n

The paper is “The Breast Cancer Epidemic: Modeling and Forecasts Based on Abortion and Other Risk
\nFactors”, by Patrick S. Carroll, published in the Journal of American Physicians and Surgeons (JPANDS).
\nBefore getting to the meat of the paper, there’s a couple of preliminary things to say about it.<\/p>\n

In an ideal world, every time you read a paper, you’d study every bit of it in great, absorbing detail. But in the real world, you can’t do that. There are too many papers; if you tried to give every paper a full and carefully detailed reading, even if you never stopped to eat and sleep,
\nyou’d be falling further behind every day. So a major skill that you acquire when you learn
\nto do research is how to triage, and decide how much attention to give to different kinds of papers.<\/p>\n

One thing that you should always consider when you set out to look at a paper is look at
\nits conclusions. In general, there are a few basic kinds of papers. There are papers presenting
\nentirely new information; there are papers that are adding something new to an established
\nconsensus; there are papers that are just piling more data onto an established consensus; and there
\nare papers that are refuting an established consensus. The way that you read a paper depends on
\nwhat kind of paper it is.<\/p>\n

If a paper is just piling on more evidence, you look at the data that it presents – and don’t pay a lot attention to much else, because they’re rehashing what’s already been said. The only really interesting thing in a paper like that is the data. So you focus your attention on the data, how it
\nwas gathered and how it was analyzed, and what (if anything) it adds to what we already know.<\/p>\n

If a paper adds something new to a consensus, then you give it more careful attention. You’re
\nstill focused primarily on the data, but you also want to carefully look at how the data was
\ngathered and analyzed, to see if the new information that they’re adding is valid.<\/p>\n

The first and last kinds of paper: the ones that present something totally new, and the ones that refute something for which there is a lot of strongly supported data, you read with
\nmuch greater care and attention to detail. These are the papers that make the strongest claims,
\nand which haven’t been carefully looked at by many different people yet, so they require the most
\ncareful attention and analysis. This paper is a member of that last class: it’s claiming to find
\na statistical link which many careful studies have not<\/em> found. So it’s in line for a very
\ncareful reading.<\/p>\n

So what’s the source? Well, it’s published in JPANDS. JPANDS in a terrible<\/em> journal. In fact, the first post on Good Math\/Bad Math was a critique of a JPANDS paper that used some of the worst statistics that I’ve ever seen published. That’s bad – the paper is appearing in a very low-credibility journal with a history of not carefully reviewing statistical analysis. That’s certainly not enough
\nto justify ignoring the paper – but the quality of the journal is a valid consideration. A paper about
\nthis topic that appears in a prestigious cancer our epidemiology journal has more credibility than
\na paper that appears in a journal known for publishing garbage. It, quite naturally, brings
\nto mind the question “Why publish this work in a non-MEDLINE indexed, low quality journal?” Like I said, it’s not enough to ignore the paper, but it does raise red flags right away: this is a paper where you’re going to have to give the data and its analysis a very careful read.<\/p>\n

So, on to the paper. What the paper does is select a set of potential risk factors for breast cancer, and then compare the incidence of those risk factors in a group of populations with the incidence of
\nbreast cancer in those same populations. That’s a sort-of strange approach. At best, that approach can
\nshow a statistical correlation, but it’s going to be a weak one – because it doesn’t maintain any link
\nbetween individuals<\/em> with risk factors and the incidence of disease. In general, you use
\na correlative study like that when you can’t<\/em> associate risk factors and incidences with specific
\nindividuals. The author does address this point: he says that it’s difficult for epidemiologists to
\nobtain information about whether a particular woman had an abortion. So that addresses that criticism, but
\nthe fact remains that it’s going to be much harder to establish a causal link rather than a correlative link using this methodology.<\/p>\n

To try to build a model, he selects a list of 7 risk factors: abortion, higher age at first live
\nbirth, childlessness, number of children, breastfeeding, hormonal contraceptive use, and hormone
\nreplacement therapy. This list raises some red flags. It omits a large number of well-known risk factors
\nwhich could easily outweigh the factors that are included in the list: smoking, alchohol, genetic risk,
\nrace. (Orac has more to say about that.) But what’s also important to notice is that these factors are
\nnot<\/em> independent. The number of women who breastfeed are, obviously, strongly correlated with the
\nnumber who’ve had children. The women who have a large number of children are much more likely to have
\ntheir first child at a younger age than the women who had only one or two children. And it ignores
\nimportant correlative factors: higher income women tend to have fewer children, later age at first birth,
\nand higher rates of breastfeeding. This list looks fishy. <\/p>\n

But what comes next is where things just totally go off the rails. He takes the 7 risk factors,
\nand using information from public health services, does a linear regression of risk factor versus cancer incidence over time. If the linear regression doesn’t produce a strong positive correlation, he throws it away<\/em>. The fact that this means that he’s asserting that well-known and well-supported
\ncorrelations should be discarded as invalid isn’t even mentioned. But what’s worse is, it’s clearly quite
\ndeliberate.<\/p>\n

On page two, he shows a graph of the data for “mean age of first live birth” plotted against breast
\ncancer risk. How does he assemble the graph for the linear regression? For each year, he takes the
\ncomplete set of women born that year. Then he computes the average age of first birth for all women born
\nthat year, and tries to correlate it with the breast cancer incidence for women born in that year. That’s
\nridiculous<\/em>. It is a completely<\/em> unacceptable and invalid use of statistics. Anyone who’s
\neven taken a college freshman course in stats should know that that is absolutely ridiculous. It’s very
\ndeliberately ignoring independence from other variables, in obviously foolish ways. I just don’t even
\nknow how to mock this, because it’s so off-the-wall ridiculous.<\/p>\n

There’s another obvious problem with the whole methodology, which pales in comparison to
\nthe dreadful way that they selected data. But I’ll mention it anyway. Linear regression and correlation
\ncoefficient measures how well a linear relationship<\/em> matches the data. It doesn’t test for
\nanything else. But there are numerous correlative and\/or causal relationships that don’t show a
\nsimple linear relationship. For example, if you look at alcohol consumption plotted against
\nvarious diseases, there’s often an initial decrease<\/em> in risk, which bottoms out and is followed by a large increase<\/em> in risk. There are often threshold effects, where something doesn’t start
\nto have an impact until beyond a minimum threshold. And so on. There’s a lot more to things that
\njust linear correlation. But all that the author considers is linear correlation. He gives no reason
\nfor that, and makes no attempt to justify it. It’s just presented as if it’s beyond question.<\/p>\n

Based on those linear regressions, he totally discards everything without a strong linear correlation
\nas being irrelevant factors that don’t need to be included in the model. That
\nleaves him with only two factors: fertility (number of live births) and abortion. So then, once
\nagain building on the assumption that linear relationships are the only things that matter, he says
\nthat they can model the breast cancer incidence via a simple linear equation:<\/p>\n

Yi<\/sub> = a + b1<\/sub>x1i<\/sub> + b2<\/sub>x2i<\/sub> + ei<\/sub><\/p>\n

In this, Yi<\/sub> is the breast cancer incidence in the group of women of age i<\/em>;
\nx1i<\/sub> is a measure of the number of abortions;\tx2i<\/sub> is a measure of fertility.
\nThey then do another linear regression using this equation to come up with coefficients for
\nthe two measured quantities. The coefficient for fertility is -0.0047, with a 95% confidence interval
\nranging from -0.0135 to _0.0041. In other words, according to their measure, fertility – the rate
\nof live birth – is not<\/em> a significant factor in breast cancer rates compared to
\nabortion.<\/p>\n

Right there, we can stop looking at the paper. When a mathematical model generates an
\nincredibly ridiculous result, something which is in direct and blatant contradiction with
\nthe known data, you throw the model right out the window, because it’s worthless. The notion that
\nabortion as a risk factor for breast cancer completely dwarfs the reduction in risk after
\nchildbirth – when we know that having children causes a dramatic decrease in the risk of
\nbreast cancer – is unquestionably wrong. If it were true, what it would mean is that the
\nnumber of cases of breast cancer among women who had no children but had an abortion (which,
\nfrom what I can estimate from data from a variety of websites is somewhere around 15%) is
\nso high<\/em> that it can completely dwarf the risk reduction among women who did
\nhave children (>80%). If that were the case, it would be incredibly obvious in the statistics
\nof breast cancer rates – you’d have a small sub-population causing an inordinately huge
\nportion of the breast cancer rates. We know that things like that are easily visible: that’s how
\nwe discovered the so-called “breast cancer genes” – a small group of women were
\ndramatically more likely to have breast cancer that the population at large.<\/p>\n

So we’ve got a model which doesn’t fit reality. What a real scientist does when this happens
\nis to say “Damn, I was wrong. Back to the ol’ drawing board”, and try to find a new model that
\ndoes<\/em> fit with reality.<\/p>\n

But not this intrepid author. He tries to handwave his way past the fact that his model is
\nwrong, by saying “The coefficient of fertility is rather small, with the 95% confidence interval straddling zero. Some improvement in breastfeeding may be offsetting fertility decline.” No, sorry, you can’t say “My mathematical model has absolutely no relation with reality, but that’s probably because one of the factors that I excluded is probably important, and so now I’m going to go on pretending that the model works.”<\/p>\n

The model is wrong<\/em>. Invalid models to not produce valid results. Stop. Do not pass go. Do not collect $200. Do not get your paper published in a decent journal. Do<\/em> get laughed at by people who aren’t clueless jackasses.<\/p>\n

At this point, we can see just why this paper appeared in a journal like JPANDS. Because it’s
\ncrap that’s just attempting to justify a political position using incredibly sloppy math; math so
\nbad that a college freshman should be able to see what’s wrong with it. But for the “reviewers” at JPANDS, apparently a college freshman level of knowledge of statistics isn’t necessary for reviewing
\na paper on statistical epidemiology.<\/p>\n","protected":false},"excerpt":{"rendered":"

My friend, fellow ScienceBlogger, and BlogFather Orac asked me to take a look at a paper that purportedly shows that abortion is a causative risk factor for breast cancer, which he posted about this morning. When the person who motivated me to start what’s turned out to be a shockingly successful blog asks for something, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[8],"tags":[],"class_list":["post-536","post","type-post","status-publish","format-standard","hentry","category-bad-statistics"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4lzZS-8E","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/536","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/comments?post=536"}],"version-history":[{"count":0,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/536\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/media?parent=536"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/categories?post=536"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/tags?post=536"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}