{"id":741,"date":"2009-02-09T11:45:30","date_gmt":"2009-02-09T11:45:30","guid":{"rendered":"http:\/\/scientopia.org\/blogs\/goodmath\/2009\/02\/09\/metric-abuse-aka-lying-with-statistics\/"},"modified":"2009-02-09T11:45:30","modified_gmt":"2009-02-09T11:45:30","slug":"metric-abuse-aka-lying-with-statistics","status":"publish","type":"post","link":"http:\/\/www.goodmath.org\/blog\/2009\/02\/09\/metric-abuse-aka-lying-with-statistics\/","title":{"rendered":"Metric Abuse – aka Lying with Statistics"},"content":{"rendered":"

I’m behind the curve a bit here, but I’ve seen and heard a bunch of
\npeople making really sleazy arguments about the current financial stimulus
\npackage working its way through congress, and those arguments are a perfect
\nexample of one of the classic ways of abusing statistics. I keep mentioning metric errors – this is another kind of metric error. The difference between this and some of the other examples that I’ve shown is that this is deliberately dishonest – that is, instead of accidentally using the wrong metric to get a wrong answer, in this case, we’ve got someone deliberately taking one metric, and pretending that it’s an entirely different metric in order to produce a desired result.<\/p>\n

As I said, this case involves the current financial stimulus package that’s working its way through congress. I want to put politics aside here: when it comes to things like this financial stimulus, there’s plenty of room for disagreement.
\nEconomic crises like the one we’re dealing with right now are really uncharted territory – they’re very rare, and the ones that we have records of have each had enough unique properties that we don’t have a very good collection of evidence
\nto use to draw solid conclusions about recoveries from them work. This isn’t like
\nphysics, where we tend to have tons and tons of data from repeatable experiments; we’re looking at a realm where there are a lot of reasonable theories, and there isn’t enough evidence to say, conclusively, which (if any) of them is correct. There are multiple good-faith arguments that propose vastly different ways of trying
\nto dig us out of this disastrous hole that we’re currently stuck in.<\/p>\n

Of course, it’s also possible to argue in bad<\/em> faith, by
\ncreating phony arguments. And that’s the subject of this post: a bad-faith
\nargument that presents real statistics in misleading ways.<\/p>\n

<\/p>\n

When you see statistics used in arguments, the number one thing that you
\nshould always be careful about is to make sure that you understand exactly<\/em> what the statistic means. One of the classic ways of misleading people with statistics is take take a valid statistic measuring some quantity, and present it as if it represented an entirely different<\/em> quantity.<\/p>\n

The current example of this comes from a variety of people who’ve been citing the work of a conservative anti-Keynesian writer named Amity Shlaes. Ms.
\nShlaes has put forward an argument that the New Deal was a failure, and that it
\ndidn’t really reduce unemployment. The full argument is in her book,
\nThe Forgotten Man: A New History of the Great Depression<\/a>\"\"; a short version of the
\nparticular claim that I’m objecting to can be found in a Wall Street Journal Op-Ed
\narticle here<\/a>.<\/p>\n

Now, in school, we’ve all seen the history of the Great Depression. We’ve all
\nheard about how the New Deal put so many people to work, doing all sorts of things – building roads, bridges, buildings, aircraft carriers, etc. All told, the WPA (the organization that the government used to fund all of the New Deal work programs) employed something over three million<\/em> people. So how did employing three
\nmillion people not<\/em> help improve the unemployment situation? It seems ridiculous on the face of it – did employing three million people somehow destroy more than<\/em> three million jobs, without anyone noticing that? <\/p>\n

The answer is that when Ms. Shlaes talks about unemployment, she’s not<\/em> really talking about unemployment. She’s careful to not<\/em> really cite unemployment rates. What she does is cite the number of permanent jobs<\/em>.<\/p>\n

The WPA programs in the New Deal were designed as a deliberate stop-gap measure. That is, there were millions of unemployed people in America, who weren’t able to afford the basic necessities, like food and housing. The point of the WPA was to get those people working, immediately, doing something to earn a wage, so that they could feed their families, pay their rent, etc. It wasn’t to give them permanent jobs working for the government, but to give them paying jobs until<\/em>
\nthe economy recovered enough that private employers would start hiring again.<\/p>\n

So in other words, Ms. Shlaes deceptively picks a statistic that excludes<\/em> the jobs created by the New Deal, and then uses that to argue that the New Deal didn’t create any jobs. If you actually state that properly, it becomes transparently ridiculous: “If you exclude all of the jobs created by the New Deal, then the New Deal didn’t create any jobs”. <\/p>\n

You can make the argument that they way the government tried to help the
\neconomy delayed that recovery. I don’t find the argument particularly convincing, but that’s probably as much my philosophical bias as anything else. There are some strong arguments that the Roosevelt economic stimulus programs were the right thing; and there are some strong arguments that it was the wrong thing. Both arguments are, largely, speculative – there just isn’t enough data about this kind of situation to be able to really know which argument is right – both arguments follow logically from their assumptions, but we don’t know which set of assumptions is right in this situation.<\/p>\n

But to argue that unemployment was not<\/em> reduced by hiring three million people? That’s idiocy. And Ms. Shlaes (and most of the people citing her) know it. In fact, she basically admits it herself, but handwaves her way past it: “To be sure, Michael Darby of UCLA has argued that make-work jobs should be counted. Even so, his chart shows that from 1931 to 1940, New Deal joblessness ranges as high as 16% (1934) but never gets below 9%. Nine percent or above is hardly a jobless target to which the Obama administration would aspire.”<\/p>\n

Read that carefully. She’s admitting<\/em> that WPA programs reduced
\nunemployment by nearly half. (And even that’s using skewed figures. Different ways of estimating unemployment during the Depression range as high as 25%.) But even in
\nthe midst of her argument about how the New Deal didn’t decrease unemployment,
\nshe’s admitting<\/em> that it reduced unemployment quite dramatically.<\/p>\n

As I said, this is a typical way of using statistics in a misleading way. Pick a statistic that measures quantity A, and use it as if it measures quantity B. You can see arguments like this all over the place.<\/p>\n

To cite another example, this time from the other side of the political spectrum: when criticizing the Bush administration’s fiscal policies, you
\nconstantly hear people talking about the Clinton surplus. They tell you that
\nunder Bill Clinton’s fiscal policies, the federal government’s budget went
\nfrom operating at a huge deficit to a huge surplus.<\/p>\n

The problem is, there was no surplus. There was never a real surplus
\nunder President Clinton. It’s once again a game of switching metrics.<\/p>\n

The government collects takes to pay for government programs. There are
\nmultiple kinds of taxes – personal income taxes, corporate taxes, capital gains
\ntaxes, inheritance taxes, social security taxes, etc. Some of the money collected
\nas those taxes goes into a general funds pool – the basic pool from which the
\ngovernment operates. Some of those taxes go into special funds, which are supposed to be used only for a specific purpose. An example of the latter kind of tax
\nis social security, which is only supposed to be used to pay social security
\nbenefits. Any excess in social security is supposed to be put away for
\nfuture retirees.<\/p>\n

When people talk about the Clinton surplus, what they mean is that by
\nthe end of Clinton’s term in office, the total<\/em> collection of funds from all<\/em> taxes during a fiscal year exceeded the total funds spent by the federal government during that fiscal year. Sounds great, right?<\/p>\n

Only the special purpose taxes, like social security, were included<\/em> in the tax collections. But in order to use social security funds for the general
\nbudget, the goverment has to borrow money<\/em> from social security. It issues
\ngovernment bonds – the exact same<\/em> bonds that make up the federal deficit. But instead of selling those bonds to private purchasers, it sells them to itself. But it’s debt, nonetheless.<\/p>\n

When someone talks about the surplus, they’re playing a misleading
\ngame by using invalid metrics. But by citing one quantity (total government income including money borrowed from social security), and pretending that it represented a different<\/em> quantity (total government income from general taxes), they can dishonestly claim to have balanced the federal budget, and produced a surplus.<\/p>\n

This tactic – using the wrong measure – is incredibly widespread, because it’s
\nincredibly effective. It’s easy to use to deceive people, because most people won’t pay attention to the exact definition used in the statistic – they’ll focus on the number. And if someone objects that the statistic is wrong, it’s easy to twist the
\nargument into a debate about the number, rather than about the meaning – which, in turn, makes it look to people observing the argument, like there’s a legitimate debate.<\/p>\n

The lesson here should be clear. Always, always make sure that you understand
\nexactly what a statistic really<\/em> measures. When someone makes an argument
\nbased on statistics, make sure that the statistics they cite really do measure what
\nthe arguer claims<\/em> they measure.<\/p>\n","protected":false},"excerpt":{"rendered":"

I’m behind the curve a bit here, but I’ve seen and heard a bunch of people making really sleazy arguments about the current financial stimulus package working its way through congress, and those arguments are a perfect example of one of the classic ways of abusing statistics. I keep mentioning metric errors – this is […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[71,8,19,40,66],"tags":[],"class_list":["post-741","post","type-post","status-publish","format-standard","hentry","category-bad-economics","category-bad-statistics","category-economics","category-metric-errors","category-unemployment"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4lzZS-bX","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/741","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/comments?post=741"}],"version-history":[{"count":0,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/741\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/media?parent=741"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/categories?post=741"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/tags?post=741"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}