Are Published Research Findings Reliable?
Submitted by charles.wallace on Sun, 08/17/2008 - 21:49.
As promoters of rational thought and critical thinking skeptics frequently rely upon published scientific research findings. But did you know that most published research findings are wrong? How can that be, and what implications does that have for the general public?
There are a variety of contributing factors that can increase the chance a published finding is false. One of the most common problems is having too small a sample size. Take for example a theoretical study that finds that there is a 50% chance a perfectly balanced coin will lands heads up when it is tossed 100 times. Now compare it to a different study that showed there is an 80% chance that a perfectly balanced coin will land heads up when tossed. However, when you read how the second study was conducted you learn the researcher only tossed a coin five times and performed one test. Which study is likely to produce a true finding? It is clear that only the first study had a large enough sample size to be trustworthy.
What other factors contribute to unreliable research findings? Conflicts of interest due to prejudice or financial interest will also decrease the likeliness that a finding is true. Conflicts of interest are rarely reported. What if, instead of a coin toss, the second study was trying to determine if an herbal supplement could help you sleep? Upon getting a favorable outcome the researchers are thrilled and publish their findings without doing any further research. Only later do independent labs find that the supplement had been contaminated with an addictive prescription drug that was responsible for the positive outcome. Or perhaps a pharmaceutical company runs six trials but only publishes the one that has a positive outcome. Does these sound far fetched? This type of thing happens more frequently than you might expect.
What does this mean for the general public? The public should be skeptical of any published research finding unless the study is well designed, large, has well defined outcomes, and is well presented. If it goes against previous research one should also wait until the results can be replicated. A single small study claiming that vitamin x will prevent you from getting a cold when flying, or that an ancient herb mixed with milk will boost your immunity, or Gemini's are more likely to be successful in love need to be treated with considerable doubt.
For a detailed examination of this topic check out, “Why Most Published Research Findings Are False,” by John P. A. Ioannidis. Or listen to the ABC Health Report.
There are a variety of contributing factors that can increase the chance a published finding is false. One of the most common problems is having too small a sample size. Take for example a theoretical study that finds that there is a 50% chance a perfectly balanced coin will lands heads up when it is tossed 100 times. Now compare it to a different study that showed there is an 80% chance that a perfectly balanced coin will land heads up when tossed. However, when you read how the second study was conducted you learn the researcher only tossed a coin five times and performed one test. Which study is likely to produce a true finding? It is clear that only the first study had a large enough sample size to be trustworthy.
What other factors contribute to unreliable research findings? Conflicts of interest due to prejudice or financial interest will also decrease the likeliness that a finding is true. Conflicts of interest are rarely reported. What if, instead of a coin toss, the second study was trying to determine if an herbal supplement could help you sleep? Upon getting a favorable outcome the researchers are thrilled and publish their findings without doing any further research. Only later do independent labs find that the supplement had been contaminated with an addictive prescription drug that was responsible for the positive outcome. Or perhaps a pharmaceutical company runs six trials but only publishes the one that has a positive outcome. Does these sound far fetched? This type of thing happens more frequently than you might expect.
What does this mean for the general public? The public should be skeptical of any published research finding unless the study is well designed, large, has well defined outcomes, and is well presented. If it goes against previous research one should also wait until the results can be replicated. A single small study claiming that vitamin x will prevent you from getting a cold when flying, or that an ancient herb mixed with milk will boost your immunity, or Gemini's are more likely to be successful in love need to be treated with considerable doubt.
For a detailed examination of this topic check out, “Why Most Published Research Findings Are False,” by John P. A. Ioannidis. Or listen to the ABC Health Report.