{"id":283,"date":"2007-01-21T19:39:04","date_gmt":"2007-01-21T19:39:04","guid":{"rendered":"http:\/\/scientopia.org\/blogs\/goodmath\/2007\/01\/21\/misrepresenting-simulations\/"},"modified":"2007-01-21T19:39:04","modified_gmt":"2007-01-21T19:39:04","slug":"misrepresenting-simulations","status":"publish","type":"post","link":"http:\/\/www.goodmath.org\/blog\/2007\/01\/21\/misrepresenting-simulations\/","title":{"rendered":"Misrepresenting Simulations"},"content":{"rendered":"

Yet another reader forwarded me a link to a rather dreadful articl<\/a>e. This one seems to be by
\nsomeone who knows better, but prefers to stick with his political beliefs rather than an honest
\nexploration of the facts.<\/p>\n

He’s trying to help provide cover for the anti-global warming cranks. Now, in light of all of the
\ndata that we’ve gathered, and all of the different kinds of analyses that have been used
\non that data, for anyone in the real world, it’s pretty undeniable that global warming is
\na real phenomena, and that at least part of it is due to humanity. <\/p>\n

<\/p>\n

One of the standard arguments from the supposed skeptics about global warming is
\nthe fact that much of our understanding of it is generated using simulations<\/em>. That’s exactly the tack that this article takes:<\/p>\n

\n

Think of a (mathematical) function. The function takes values as input, processes the values, and spits out a value. A simulation is a lot like a function, with three crucial exceptions. First, a simulation has no real input values, but uses instead simulated input. Second, because the input is simulated, instead of running the simulation only once (as you would with a function), you have to run a simulation many times (iterations), then statistically analyze the output — the simulated output, calculated from the simulated input. Third, because the output of the simulation is simulated output and there are multiple outputs (because the simulation must be run many times), it must be statistically analyzed for reliability and the multiple results must be analyzed statistically.<\/p>\n

A function is certain. A simulation is uncertain.<\/p>\n<\/blockquote>\n

This is utter bullshit. No nice way to put it; it’s utter crap. It is not<\/em> a fact that
\nsimulations don’t take real<\/em> input data. It is also not<\/em> a fact that all simulations
\nneed to be run multiple times. This is a pure strawman; some<\/em> simulations are run with
\nsimulated input; some<\/em> simulations are run multiple times with varying inputs to get a sense
\nof trends. It is manifestly not<\/em> the fact that all<\/em> simulations work this way; and
\nmany of the simulations used for analyzing global warming trends were fully deterministic simulations run using real<\/em> measurements. <\/p>\n

Let’s take a moment and ask the fundamental question: What is a simulation?<\/p>\n

For our purposes: a simulation is a computer program which implements a mathematical model
\nof some phenomenon. The input to a simulation is the data needed to describe an initial state
\nof that phenomena. The simulation is run against the input data, and produces a description of
\nwhat it’s mathematical model predicts about the state of the phenomena at subsequent points in time.<\/p>\n

Given a simulation model, it can be run in a number of different ways:<\/p>\n

    \n
  1. Verification: given a set of data for some point in the past, the simulation can be run
    \nto see if its results match the present.<\/li>\n
  2. Prediction: given a set of data for the present, the simulation can be run to
    \ngenerate predictions about the future.<\/li>\n
  3. Exploration: given a manufactured<\/em> set of data, the simulation can be run
    \nto explore what could happen in a given situation, or to study how the model works
    \nin various cases.<\/li>\n<\/ol>\n

    The author of the original article pretends that everything is case 3 – the exploratory case,
    \nwhere the data being input is manufactured, rather than being a representation of real measurements.
    \nHe also assumes that the simulation is stochastic<\/em>: that is, that it is using some sort of
    \nrandomness in its input, so that starting from the same<\/em> input data, the simulation will
    \ngenerate different results. In my<\/em> experience, that’s incredibly rare in computer simulations
    \n– in fact, I can’t recall ever seeing a single<\/em> simulation that wasn’t fully deterministic –
    \nmeaning that it always generates the same result for the same input.<\/p>\n

    There’s one thing he got right: functions are certain. Given a function, you know<\/em> what
    \nits result is, and you can check whether or not the result is correct. For a simulation, the result
    \nis more fuzzy – the simulation may be generating a “correct” result in the sense that it doesn’t have
    \nany bugs, and it generates the result that its model says it should; and at the same time be
    \ncompletely wrong<\/em> because the mathematical model that it’s using doesn’t accurately
    \nrepresent reality. And there’s one other thing that’s true, although he doesn’t explicitly mention it: many simulations are probabilistic, in the sense that instead of generating exactly one results, they compute multiple possibilities: when there are choices about what to do in the model, they
    \nrun all of them, generating a probability for each of the branches.<\/p>\n

    With that out of the way, let’s continue on to the next interesting part of the article.<\/p>\n

    \n

    So how does a simulation simulate input values? Usually, by taking real data, analyzing it statistically to determine its frequency and distribution, then using statistics to generate input values using the same frequency and distribution. Note that in order for this to work, one must assume that the data are stable, that is, that the frequency and distribution of the data will not change over time.<\/p>\n

    The part of the simulation that corresponds to a function is known as the model. Obviously, only an accurate model can produce reliable results (output), and the more accurate the model, the more reliable the results.<\/p>\n

    Simulations can be powerful tools for making predictions, and are used in many fields, including
    \nbusiness. However, because simulations use simulated (that is, not real) input and result in
    \nsimulated (that is, not real) output, they have no evidentiary power — that is, you cannot use a
    \nsimulation as evidence for anything, nor can you call the output of a simulation (real) data.<\/p>\n<\/blockquote>\n

    First paragraph is not nonsense, but it’s not exactly honest<\/em>, either. He wants to make
    \nit look like the models are as uncertain as possible, so he stresses the idea of a stochastic model.
    \nA stochastic model is entirely probabilistic – its model is based on nothing but statistics about how
    \nthings have behaved before. Stochastic models are relatively weak, as models go; they’re generally
    \nused in cases where we don’t have a good behavioral model that accurately models the real phenomena
    \nthat’s being simulated. In any situation where we use simulations, we strongly<\/em> prefer
    \nphysical models – that is, instead of using stochastic models, we prefer a simulation that’s based on
    \nreally simulating physical behaviors, rather than just playing with probabilities. What he’s saying is misleading – because he’s pretending that all<\/em> models are stochastic.<\/p>\n

    He also<\/em> deliberately overstates the case about the evidentiary value of simulations. A simulation is never considered the equivalent of real-world evidence in terms of quality; but they
    \nmost emphatically are<\/em> frequently used as supporting evidence for various theories. The quality of a simulation as evidence is generally based on how well it performs in verification runs – a simulation that can be demonstrated to generate accurate results in a wide variety of situations is considered a good piece of supporting evidence when run on data similar to the data used for the verifications. <\/p>\n

    For example, the US Army’s ordinance testing facility in Maryland now uses simulations
    \nfor most<\/em> of its tests. It uses tests for information gathering, and it periodically
    \nperforms tests to validate the simulations; but the quality of the simulations has gotten high enough that for many purposes, they no longer consider it necessary to run real tests of dropping explosives from an airplane. They quite definitely consider the results of those simulations to be evidence!<\/p>\n

    And now, finally, we get to the sleaziest part.<\/p>\n

    \n

    You can use estimated data in your model, but doing so inserts another layer of uncertainty into the simulation results. Let me explain.<\/p>\n

    Since the data in our model are estimated, we must use statistics to determine their validity. Since Mr. Lewis uses dice, I will as well, sticking for the sake of simplicity and clarity to rolling one die.<\/p>\n

    If you roll a die, the probability that you will roll, say, a 1 is 1\/6. If you roll the die a second time, the probability that you will roll, say, a second 1 is 1\/6 * 1\/6, or 1\/36. If you roll the die a third time, the probability that you will roll, say, a third 1 is 1\/6 * 1\/6 * 1\/6, or 1\/216, and so forth.<\/p>\n

    Estimated variables in a simulation model must be treated in the same way as rolling a die, because each is uncertain, and each involves probability. Assuming that our climatologist is ethical, each of the estimated variables in his model should fall within the statistical norm of reliability, or be 95% reliable. Given the complexity of climatological models, hundreds of such estimated variables would be necessary, but for clarity’s sake, we will say the model includes only 50 such variables.<\/p>\n

    That means that the reliablility of the simulation model is 0.95^50 (0.95 raised to the fiftieth power), or 0.0769, or 7.7%. So even if we didn’t have the uncertainty of the simulated input (not to mention the additional uncertainty of assuming that the data are stable), even if there were no uncertainty in our ouputs themselves, our simulation results would only 7.7% reliable.\t<\/p>\n<\/blockquote>\n

    This is a thoroughly<\/em> dishonest bunch of babble, which is in no way<\/em> an accurate description of anything<\/em>. <\/p>\n

    Even if we accept what he says at face value: that there are multiple variables
    \nin a simulation which need to be considered separately in terms of probability – he’s quite
    \ndeliberately ignoring the correct<\/em> way of combining those probabilities. In fact, he’s really just trying to play the inverse of a classic “big numbers” game – he wants to artificially combine things to make the probability look as untrustworthy as possible. The trick is in pretending
    \nthat all 50 (or whatever) variables in the simulation are independent<\/em>. In real
    \nclimate simulations, the kinds of things that become variables are not<\/em> independent. To give a couple of examples, real climatalogical simulations will include a parameter to describe the
    \nhumidity of airmasses based on temperature; and simulation to describe the viscosity of airmasses based on temperature and humidity. Those are not<\/em> independent – the viscosity of the airmass is determined in part by its humidity; the ability of the airmass to pick up more moisture while over the ocean is determined in part by its viscosity. The probabilities of these things being correct are not independent – if one is right, the other is almost certainly right; and if one is wrong, the other is almost certainly wrong – because each depends on the correctness of the other. Dependent variables get treated very<\/em> differently in a probability calculation that independent variables – that’s what     Bayes theorem<\/a> is all about.<\/p>\n

    But it’s much worse than just making a misleading probability argument. He’s very deliberately
    \nmischaracterizing how we model the accuracy of a simulation. The accuracy of simulation is based on
    \nits performance and the known accuracy of the fundamental model which it’s based on. So, for example,
    \nmost airplane manufacturers no longer use wind tunnels – computational fluid dynamics simulations
    \ngenerate better<\/em> results than the wind tunnel (Do a websearch on “Boeing” and “Tranair”). The
    \nreliability of the simulation is based on two things. One is a long history of measuring things on
    \ninstrumented aircraft, and comparing the measurements to the predictions from the simulations; the
    \nother is the known accuracy of the Navier Stokes equations, and the computational methods used to
    \nimplement NS systems. On the basis of those two, we come up with results about how accurate we
    \nbelieve the models to be. <\/p>\n

    And further – we look at simulations based on multiple<\/em> models. If 20 different models, generated in 20 different ways, all of which have strong track records for accuracy – if all 20 of them have been been implemented by simualations whose quality has been demonstrated – and all of them generate nearly the same result, and no<\/em> system\/model with a proven track record disagrees, then we consider the results of those simulations to be very strong<\/em> evidence. <\/p>\n

    Of course, he saves the worse for last.<\/p>\n

    \n

    Climatological simulations cannot be taken very seriously. They can certainly never be taken as evidence or proof, as no simulation can be, because they are simulations. They aren’t real.<\/p>\n

    What disturbs me about all this global warming warfare is that the climatologists know this. They know that their models have no evidentiary power. Yet, they disingenuously claim the reverse. This isn’t science. It’s politics. It’s dishonest. And it’s a breach of professional ethics and integrity.<\/p>\n<\/blockquote>\n

    So says the man who just threw together a bundle of lies and misrepresentations to try to
    \nsupport a pre-determined opinion without regard for the facts. And he accuses others<\/em>
    \nof dishonesty and breaches of ethics and integrity.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Yet another reader forwarded me a link to a rather dreadful article. This one seems to be by someone who knows better, but prefers to stick with his political beliefs rather than an honest exploration of the facts. He’s trying to help provide cover for the anti-global warming cranks. Now, in light of all of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[5],"tags":[],"class_list":["post-283","post","type-post","status-publish","format-standard","hentry","category-bad-physics"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4lzZS-4z","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/283","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/comments?post=283"}],"version-history":[{"count":0,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/283\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/media?parent=283"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/categories?post=283"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/tags?post=283"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}