{"id":102,"date":"2006-08-06T12:39:48","date_gmt":"2006-08-06T12:39:48","guid":{"rendered":"http:\/\/scientopia.org\/blogs\/goodmath\/2006\/08\/06\/qa-what-is-information\/"},"modified":"2006-08-06T12:39:48","modified_gmt":"2006-08-06T12:39:48","slug":"qa-what-is-information","status":"publish","type":"post","link":"http:\/\/www.goodmath.org\/blog\/2006\/08\/06\/qa-what-is-information\/","title":{"rendered":"Q&A: What is information?"},"content":{"rendered":"

I received an email from someone with some questions about information theory; they relate to some sufficiently common questions\/misunderstandings of information theory that I thought it was worth turning the answer into a post.
\nThere are two parts here: my correspondent started with a question; and then after I answered it, he asked a followup.
\nThe original question:
\n————————
\n>Recently in a discussion group, a member posted a series of symbols, numbers,
\n>and letters:
\n>
\n>`+
\n>The question was what is its meaning and whether this has information or not.
\n>I am saying is does have information and an unknown meaning (even though I am
\n>sure they were just characters randomly typed by the sender), because of the
\n>fact that in order to recognize a symbol, that is information. Others are
\n>claiming there is no information because there is no meaning. Specifically that
\n>while the letters themselves have meaning, together the message or “statement”
\n>does not, and therefore does not contain information. Or that the information
\n>content is zero.
\n>
\n>Perhaps there are different ways to define information?
\n>
\n>I think I am correct that it does contain information, but just has no meaning.
\nThis question illustrates one of the most common errors made when talking about information. Information is *not* language: it does *not* have *any* intrinsic meaning. You can describe information in a lot of different ways; in this case, the one that seems most intuitive is: information is something that reduces a possibility space. When you sit down to create a random string of characters, there is a huge possibility space of the strings that could be generated by that. A specific string narrows the space to one possibility. Anything that reduces possibilities is something that *generates* information.
\nFor a very different example, suppose we have a lump of radium. Radium is a radioactive metal which breaks down and emits alpha particles (among other things). Suppose we take out lump of radium, and put it into an alpha particle detector, and record the time intervals between the emission of alpha particles.
\nThe radium is *generating information*. Before we started watching it, there were a huge range of possibilities for exactly when the decays would occur. Each emission – each decay event – narrows the possibility space of other emissions. So the radium is generating information.
\nThat information doesn’t have any particular *meaning*, other than being the essentially random time-stamps at which we observed alpha particles. But it’s information.
\nA string of random characters may not have any *meaning*; but that doesn’t mean it doesn’t contain information. It *does* contain information; in fact, it *must* contain information: it is a *distinct* string, a *unique* string – one possibility out of many for the outcome of the random process of generation; and as such, it contains information.
\nThe Followup
\n—————
\n>The explanation I have gotten from the person I have been debating, as to what
\n>he says is information is:
\n>
\n>I = -log2<\/sub> P(E)
\n>
\n>where:
\n>I: information in bits
\n>P: probability
\n>E: event
\n>
\n>So for the example:
\n>`+
\n>He says:
\n>”I find that there is no meaning, and therefore I infer no information. I
\n>calculate that the probability of that string occuring was ONE (given no
\n>independent specification), and therefore the amount of information was ZERO. I
\n>therefore conclude it has no meaning.”
\n>
\n>For me, even though that string was randomly typed, I was able to look at the
\n>characters, and find there placement on a QWERTY keyboard, compare the
\n>characters to the digits on the hand, and found that over all, the left hand
\n>index finger keys were used almost twice as much. I could infer that a person
\n>was left handed tried to type the “random” sequence. So to me, even though I
\n>don’t know the math, and can’t measure the amount of information, the fact I
\n>was able to make that inference of a left handed typer tells me that there is
\n>information, and not “noise”.
\nThe person quoted by my correspondent is an idiot; and clearly one who’s been reading Dembski or one of his pals. In my experience, they’re the only ones who continually stress that log-based equation for information.
\nBut even if we ignore the fact that he’s a Dembski-oid, he’s *still* an idiot. You’ll notice that nowhere in the equation does *meaning* enter into the definition of information content. What *does* matter is the *probability* of the “event”; in this case, the probability of the random string of characters being the result of the process that generated it.
\nI don’t know how he generated that string. For the sake of working things through, let’s suppose that it was generated by pounding keys on a minimal keyboard; and let’s assume that the odds of hitting any key on that keyboard are equal (probably an invalid assumption, but it will only change the quantity of information, not its presence, which is the point of this example.) A basic minimal keyboard has about sixty keys. (I’m going by counting the keys on my “Happy Hacking Keyboard”. Of those, seven are shifts of various sorts: 2 shift, 2 control, one alt, one command, one mode), and one is “return”. So we’re left with 52 keys. To make things simple, let’s ignore the possibility of shifts affecting the result (this will result in us getting a *lower* information content, but that’s fine). So, it’s 80 characters; each *specific* character generated is an event with probability 1\/52. So each *character* of the string has, by the Shannon-based definition quoted above, about 5.7 bits of information. The string as a whole has about 456 bits of information.
\nThe fact that the process that generated it is random doesn’t make the odds of a particular string be one. For a trivial example, I’m going to close my eyes and pound on my keyboard with both hands twice, and then select the first 20 characters from each:
\nFirst: “`;kldsl;ksd.z.md\u03a9.l.x`”
\nSecond: “`lkficeewrflk;erwm,.r`”
\nQuite different results overall? Seems like I’m a bit heavy on the right hand on the k\/l area. But note how different the outcomes are. *That* is information. Information without *semantics*; without any intrinsic meaning. But that *doesn’t matter*. Information *is not language*; the desire of [certain creationist morons][gitt] to demand that information have the properties of language is nonsense, and has *nothing* to do with the mathematical meaning of information.
\nThe particular importance of this fact is that it’s a common creationist canard that information *must* have meaning; that therefore information can’t be created by a natural process, because natural processes are random, and randomness has no meaning; that DNA contains information; and therefore DNA can’t be the result of a natural process.
\nThe sense in which DNA contains information is *exactly* the same as that of the random strings – both the ones in the original question, and the ones that I created above. DNAs information is *no different*.
\n*Meaning* is something that you get from language; information *does not* have to be *language*. Information *does not* have to have *any* meaning at all. That’s why we have distinct concepts of *messages*, *languages*, *semantics*, and *information*: because they’re all different.
\n[gitt]: http:\/\/scienceblogs.com\/goodmath\/2006\/07\/bad_bad_bad_math_aig_and_infor.php<\/p>\n","protected":false},"excerpt":{"rendered":"

I received an email from someone with some questions about information theory; they relate to some sufficiently common questions\/misunderstandings of information theory that I thought it was worth turning the answer into a post. There are two parts here: my correspondent started with a question; and then after I answered it, he asked a followup. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[30],"tags":[],"class_list":["post-102","post","type-post","status-publish","format-standard","hentry","category-information-theory"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4lzZS-1E","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/102","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/comments?post=102"}],"version-history":[{"count":0,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/102\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/media?parent=102"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/categories?post=102"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/tags?post=102"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}