Monday, April 29, 2013

Bad math in BBC

In the BBC Magazine story Amanda Knox and bad math in court, there is an explanation of probability relating to evidence in a legal trial.

The article quotes mathematician Coralie Colmez, co-author of "Math on Trial: How numbers get used and abused in the courtroom". Well, I hope they misquoted her. The math presented in the article does not appear to add up.

The example given is:

"You do a first test and obtain nine heads and one tail... The probability that the coin is fair given this outcome is about 8%, [and the probability] that it is biased, about 92%. Pretty convincing, but not enough to convict your coin of being biased beyond a reasonable doubt," Colmez says.

Technically in this situation we talk about the probability that a fair coin will give a particular outcome. We can calculate the chance of a fair coin giving nine or more heads (or zero or one tails), using the binomial distribution.

In Excel we can use =BINOMDIST(1,10,0.5,TRUE). In the statistical program R the formula is pbinom(1,10,0.5).

The answer is 0.01074219, or about 1%. I don't know where the 8% quoted in the article comes from.

The more subtle problem is that the chance of a fair coin producing a result is not the same as the chance of the coin being fair given a result. This is what the article seems to imply, and is a common mistake.

The theory we need for this kind of statement is Bayes' Theorem, which deals with conditional probabilities:

P(A|B) = P(B|A) P(A) / P(B)

Where P(A|B) means "Probability of A, given B".

To know the probability of this particular coin being fair after an experiment we'd need to know more information about the whole population of coins.