The New Scientist reports on a scientific article by Dr. J.P. Ioannidis claiming that most scientific papers are wrong, and tries to explain why. I’ll let you read the story by yourselves but I’d like to point out some aspects that I thought were self-evident, but that’s just because I’m a statistician.
First of all, Ioannidis seems very fond of blaming the 5% significance level of a statistical test. Well, everybody should. Actually, it would be pretty stupid to think that every single hypothesis in the world could be proven with a significance level of 5%. If you give a student 20 exams on the same subject, the probability is fairly high that he will fail on at least one of these exams, regardless of his skill level. When doing a statistical experiment, that is to say comparing a null hypothesis to an alternative, there are at least 4 things you should consider before anything else:
- The significance level you want: how sure do you want to be of your findings, or what risk you are willing to take that you make a mistake when you choose the alternative over the null hypothesis
- The power you want to reach: to what extent should you be able to choose the alternative, in other words what risk you are willing to take that you make a mistake when choosing the null hypothsis instead of the alternative
- The difference you want to be able to detect: that is to say the time when you want to be able to reject the null hypothsis
- The distribution of the test statistic under both the null and alternative hypotheses: this will be useful later on and it also crystallizes what you will be doing after your experiment
You will notice I haven’t talked about sample size. That’s because sample size is dependent on these four factors. Most researchers meet with a statistician the first time by saying: “I have ‘n’ individuals and I want to test this…” They usually have never considered the detectable difference or the power of the experiment, but they hold on to a 5% significance level. This is especially dramatic when you consider genetic microarray analyses. These chips can contain ten thousand experiments each and everyone thinks a 5% significance level is okay! And that’s just one of the problems with microarrays.
The other problem is simultaneous testing of hypotheses. Let’s say you have three kinds of aspririn and you want to see if there is a difference between their efficiency. You could check brand A vs. B, A vs. C and B vs. C; because you have no idea which ones are different and which ones aren’t. Well, it could actually happen that, at the 5% significance level, you’d find A is better than B, C is better than A and B is better than C. Think it over, you’ve made a mistake somewhere.
The mistake ? Well, you shouldn’t have taken a 5% significance level. One idea is rather to take 0.05/3 as a significance level for each comparison. What happens if you take a microarray with 10000 comparisons ? Do you take a 0.05/10000 significance level ? (For those who are considering this I’ll save you the trouble: don’t!)
Finally, the subject is wide and it is troubling to think that some of the other considerations Dr. Ioannidis makes are actually common in research. One last advice: before you experiment, meet a statistician. They’re friendly people with a wide array of interests and they’ll save you a load of trouble later on in your life.
August 31st, 2005 | General Science | No comments
Light won’t be only for illuminating a room soon.