To understand a lot of statistical ideas, you need to know about
\nprobability. The two fields are inextricably entwined: sampled statistics
\nworks because of probabilistic properties of populations.<\/p>\n
I approach writing about probability with no small amount of trepidation.<\/p>\n
For some reason that I’ve never quite understood, discussions of probability
\ntheory bring out an intensity of emotion that is more extreme than anything else
\nI’ve seen in mathematics. It’s an almost religious topic, like programming
\nlanguages in CS. This post is intended really as a flame attractor: that is, I’d request that if you want to argue about Bayesian probability versus frequentist probability, please do it here, and don’t clutter up every comment thread that
\ndiscusses probability! <\/p>\n
There are two main schools of thought in probability:
\nfrequentism and Bayesianism, and the Bayesians have an intense contempt for the
\nfrequentists. As I said, I really don’t get it: the intensity seems to be mostly
\none way – I can’t count the number of times that I’ve read Bayesian screeds about
\nthe intense stupidity of frequentists, but not the other direction. And while I
\nsit out the dispute – I’m undecided; sometimes I lean frequentist, and sometimes I
\nlean Bayesian – every time I write about probability, I get emails and comments
\nfrom tons of Bayesians tearing me to ribbons for not being sufficiently
\nBayesian.<\/p>\n
It’s hard to even define probability without getting into trouble, because the
\ntwo schools of thought end up defining it quite differently.<\/p>\n
The frequentist approach to probability basically defines probability in terms The bayesian approach is based on incomplete knowledge. It says that you only Like I said, I tend to sit in the middle. On the one hand, I think that the
\nof experiment. If you repeated an experiment an infinite number of times, and
\nyou’d find that out of every 1,000 trials, a given outcome occured 350 times, then
\na frequentist would say that the probability of that outcome was 35%. Based on
\nthat, a frequentist says that for a given event, there is a true<\/em>
\nprobability associated with it: the probability that you’d get from repeated
\ntrials. The frequentist approach is thus based on studying the “real” probability
\nof things – trying to determine how close a given measurement from a set of
\nexperiments is to the real probability. So a frequentist would define probability
\nas the mathematics of predicting the actual likelihood of certain events occuring
\nbased on observed patterns.<\/p>\n
\nassociate a probability with an event because there is uncertainty about it –
\nbecause you don’t know all the facts. In reality, a given event either will happen
\n(probability=100%) or it won’t happen (probability=0%). Anything else is an
\napproximation based on your incomplete knowledge. The Bayesian approach is
\ntherefore based on the idea of refining predictions in the face of new knowledge.
\nA Bayesian would define probability as a mathematical system of measuring the
\ncompleteness of knowledge used to make predictions. So to a Bayesian, strictly speaking, it’s incorrect to say “I predict that there’s a 30% chance of P”, but rather “Based on the current state of my knowledge, I am 30% certain that P will occur.”<\/p>\n
\nBayesian approach makes some things clearer. For example, a lot of people
\nfrequently misunderstand how to apply statistics: they’ll take a study showing
\nthat, say, 10 out of 100 smokers will develop cancer, and assume that it means
\nthat for a specific smoker, there’s a 10% chance that they’ll develop cancer.
\nThat’s not true. The study showing that 10 out of 100 people who smoke will develop cancer can be taken as a good starting point for making a prediction – but a Bayesian will be very clear on the fact that it’s incomplete knowledge, and that it therefore isn’t very meaningful unless you can add more information to increase the certainty.<\/p>\n