{"id":2219,"date":"2013-08-24T20:32:04","date_gmt":"2013-08-25T00:32:04","guid":{"rendered":"http:\/\/scientopia.org\/blogs\/goodmath\/?p=2219"},"modified":"2013-08-24T20:32:04","modified_gmt":"2013-08-25T00:32:04","slug":"kolmogorovs-axioms-of-probability","status":"publish","type":"post","link":"http:\/\/www.goodmath.org\/blog\/2013\/08\/24\/kolmogorovs-axioms-of-probability\/","title":{"rendered":"Kolmogorov&#039;s Axioms of Probability"},"content":{"rendered":"<p> The way that I&#8217;ve talked about probability so far is mostly informal. That&#8217;s the way that probability theory was treated for a long time. You defined probability spaces over collections of equal probability sets. You combined probability spaces by combining their events into other kinds of equally probable events.<\/p>\n<p> The problem with that should be obvious: it&#8217;s circular. You want to define the probability of events; to do that, you need to start with equally probable events, which means that on some level, you already know the probabilities. If you don&#8217;t know the probabilities, you can&#8217;t talk about them. The reality is somewhat worse than that, because this way of looking at things completely falls apart when you start trying to think about infinite probability spaces!<\/p>\n<p> So what can you do?<\/p>\n<p> The answer is to reformulate probability. Mathematicians knew about this kind of problem for a very long time, but what they mostly just ignored it: probability wasn&#8217;t considered a terribly interesting field.<\/p>\n<p> Then, along came Kolmogorov &#8211; the same brilliant guy who&#8217;s theory of computational complexity is so fascinating to me! Kolmogorov created a new formulation of probability theory. Instead of starting with a space of equally probable discrete events, you start with a <em>measure space<\/em>.<\/p>\n<p> Before we can look at how Kolmogorov reformulated probability (the <em>Kolmogorov axioms<\/em>), we need to look at just what a measure space is.<\/p>\n<p> A measure space is just a set with a <em>measure function<\/em>. So let <b>X<\/b> be a set. A <em>measure<\/em> &mu; on <b>X<\/b> is a function from a subset of <b>X<\/b> to a real number: <img src='http:\/\/l.wordpress.com\/latex.php?latex=mu%3A%202%5EX%20rightarrow%20R&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='mu: 2^X rightarrow R' style='vertical-align:1%' class='tex' alt='mu: 2^X rightarrow R' \/> with the following properties:<\/p>\n<ul>\n<li> Measures are non-negative: <img src='http:\/\/l.wordpress.com\/latex.php?latex=forall%20x%20subseteq%20X%3A%20mu%28x%29%20ge%200&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='forall x subseteq X: mu(x) ge 0' style='vertical-align:1%' class='tex' alt='forall x subseteq X: mu(x) ge 0' \/><\/li>\n<li> The measure of the empty set is always 0: <img src='http:\/\/l.wordpress.com\/latex.php?latex=mu%28emptyset%29%20%3D%200&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='mu(emptyset) = 0' style='vertical-align:1%' class='tex' alt='mu(emptyset) = 0' \/><\/li>\n<li> The measure of a finite sequence of unions is the sum of the individual measures<img src='http:\/\/l.wordpress.com\/latex.php?latex=mu%28x%20%2B%20y%29%20%3D%20mu%28x%29%20%2B%20mu%28y%29&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='mu(x + y) = mu(x) + mu(y)' style='vertical-align:1%' class='tex' alt='mu(x + y) = mu(x) + mu(y)' \/><\/li>\n<\/ul>\n<p> So the idea is pretty simple: a measure space is just a way of defining the <em>size<\/em> of a subset in a consistent way.<\/p>\n<p> To work with probability, you need a measure space where the measure of the entire set is 1. With that idea in mind, we can put together a proper, formal definition of a probability space that will really allow us to work with, and to combine probabilities in a rigorous way.<\/p>\n<p> Like our original version, a probability space has a set of events, called its <em>event space<\/em>. We&#8217;ll use <b>F<\/b> to represent the set of all possible events, and <em>e<\/em> to represent an event in that set.<\/p>\n<p> There are three fundamental axioms of probability, which are going to look <em>really<\/em> similar to the three axioms of a measure space:<\/p>\n<ol>\n<li> Basic measure: the probability of any event is a positive real number: <img src='http:\/\/l.wordpress.com\/latex.php?latex=forall%20e%20in%20E%3A%20P%28e%29%20%3E%3D%200&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='forall e in E: P(e) >= 0&#8242; style=&#8217;vertical-align:1%&#8217; class=&#8217;tex&#8217; alt=&#8217;forall e in E: P(e) >= 0&#8242; \/>.<\/li>\n<li> Unit measure: the probability that <em>some<\/em> event will occur is 1, which we write as <img src='http:\/\/l.wordpress.com\/latex.php?latex=P%28Omega%29%3D1&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='P(Omega)=1' style='vertical-align:1%' class='tex' alt='P(Omega)=1' \/> (<img src='http:\/\/l.wordpress.com\/latex.php?latex=Omega&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='Omega' style='vertical-align:1%' class='tex' alt='Omega' \/> is called the <em>unit event<\/em>, and is the union of all possible events.) Alternatively, the probability of no event occurring is 0: <img src='http:\/\/l.wordpress.com\/latex.php?latex=P%28emptyset%29%3D0&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='P(emptyset)=0' style='vertical-align:1%' class='tex' alt='P(emptyset)=0' \/>.<\/li>\n<li>Combination: For any two distinct events or sets of events <img src='http:\/\/l.wordpress.com\/latex.php?latex=e&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='e' style='vertical-align:1%' class='tex' alt='e' \/> and <img src='http:\/\/l.wordpress.com\/latex.php?latex=f&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='f' style='vertical-align:1%' class='tex' alt='f' \/>, the probability of <img src='http:\/\/l.wordpress.com\/latex.php?latex=e&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='e' style='vertical-align:1%' class='tex' alt='e' \/> or <img src='http:\/\/l.wordpress.com\/latex.php?latex=f&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='f' style='vertical-align:1%' class='tex' alt='f' \/> is <img src='http:\/\/l.wordpress.com\/latex.php?latex=P%28e%29%20%2B%20P%28f%29&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='P(e) + P(f)' style='vertical-align:1%' class='tex' alt='P(e) + P(f)' \/>: <img src='http:\/\/l.wordpress.com\/latex.php?latex=forall%20e%2C%20f%20subseteq%20P%3A%20e%20cap%20f%20%3D%20emptyset%20Rightarrow%20P%28e%20cup%20%20f%29%20%3D%20P%28e%29%20%2B%20P%28f%29&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='forall e, f subseteq P: e cap f = emptyset Rightarrow P(e cup  f) = P(e) + P(f)' style='vertical-align:1%' class='tex' alt='forall e, f subseteq P: e cap f = emptyset Rightarrow P(e cup  f) = P(e) + P(f)' \/>. This can be extended to any <em>countable<\/em> sequence of unions.<\/li>\n<\/ol>\n<p> This is very similar to the informal version we used earlier. But as we&#8217;ll see later, this simple formulation from measure theory will give us a lot of additional power.<\/p>\n<p> It&#8217;s worth taking a moment to point out two implications of these axioms. (In fact, I&#8217;ve seen some presentations that treat some of these as additional axioms, but they&#8217;re provable from the first three.<\/p>\n<ul>\n<li> Monotonicity: if <img src='http:\/\/l.wordpress.com\/latex.php?latex=e%20subeq%20f&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='e subeq f' style='vertical-align:1%' class='tex' alt='e subeq f' \/>, then <img src='http:\/\/l.wordpress.com\/latex.php?latex=P%28e%29%20le%20P%28f%29&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='P(e) le P(f)' style='vertical-align:1%' class='tex' alt='P(e) le P(f)' \/>.<\/li>\n<li> Upper Bound: for any event or set of events <img src='http:\/\/l.wordpress.com\/latex.php?latex=e&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='e' style='vertical-align:1%' class='tex' alt='e' \/>, <img src='http:\/\/l.wordpress.com\/latex.php?latex=P%28e%29%20ge%200%20land%20P%28e%29%20le%201&#038;bg=FFFFFF&#038;fg=000000&#038;s=0' title='P(e) ge 0 land P(e) le 1' style='vertical-align:1%' class='tex' alt='P(e) ge 0 land P(e) le 1' \/>.<\/li>\n<\/ul>\n<p> The brilliance of Kolmogorov was realizing that these rules were <em>everything you need<\/em> to work out any probability you want &#8211; in both finite and infinite spaces. We&#8217;ll see that there&#8217;s a lot of complexity in the combinatorics of probability, but it will all always ultimately come back to these three rules.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The way that I&#8217;ve talked about probability so far is mostly informal. That&#8217;s the way that probability theory was treated for a long time. You defined probability spaces over collections of equal probability sets. You combined probability spaces by combining their events into other kinds of equally probable events. The problem with that should be [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[53],"tags":[],"class_list":["post-2219","post","type-post","status-publish","format-standard","hentry","category-probability"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4lzZS-zN","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/2219","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/comments?post=2219"}],"version-history":[{"count":0,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/posts\/2219\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/media?parent=2219"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/categories?post=2219"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.goodmath.org\/blog\/wp-json\/wp\/v2\/tags?post=2219"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}