As my attempt to catch up on TED talks continues, this one appealed to my geeky statistical background (even if much of it is now forgotten). Oxford math prof Peter Donnelly explains why seemingly logical conclusions, especially those made by supposed 'experts', can be completely wrong...as anyone who's ever been at the wrong end of quant pre-testing will tell you (ah, the joys of cheap jibes!).
(Note: there seems to be some probs with TED's embedded videos, so you might want to jump over to their site if it doesn't load properly)
I found one example of reading data wrongly that Peter used particularly intriguing, and worthy of repeating (albeit with a different context that better fits what we all get up to).
So (if you're ready for some number action) picture the scene...
- You are testing a new product which (for sake of argument) needs 20% penetration to be viable.
- To simplify things, we are going to say that claimed liking in testing is a good predictor of penetration (whether it is or not is irrelevant to the statistical point) - i.e. you need a liking score of 20% to guarantee success.
- And to simplify things further still, assume a polarising product - you either like it or you don't.
- 1 million people will be interviewed (could be more if you like - expense is no issue as we have a bottomless research budget), to minimise sampling error.
- And the test you will be using is known to be 90% accurate.
When the results come back, you find 204,000 people said they liked the product (or 20.4%…exactly what you need). Remembering the 90% accuracy of the test (which is pretty high), should you go ahead?
Now obviously this is a trick question, and you know the answer is no.
But what is interesting (and central to the data logic trap) is the fact that, if you had gone ahead, it wouldn't have a just been a minor problem, but a disaster - you wouldn't have missed your penetration target by just a little, you would have missed it by over a third.
The reason? Because the test (despite being 90% accurate overall) was actually less than 60% accurate at predicting liking scores.
And here's why...
In reality, only 130,000 people liked the product (or 13% penetration). Based on 90% test accuracy, 117,000 of these would have rated correctly, with 13,000 giving a false negative. Conversely, some 870,000 disliked the product. Again, applying 90% accuracy, 783,000 rated correctly...but 87,000 (or 43% of 'positive' responders) actually gave a false positive.
Hence the fact that the test, though 90% accurate, can also be only 60% correct at predicting the measure that really matters (it is, coincidently, 98% accurate at predicting dislike). And why it can be dangerous to take seemingly logical and objective data-driven conclusions at face value.
Nerdy but worrying!
Recent Comments