Introduction    Types of Mistakes       Suggestions       Resources       Table of Contents     About    Glossary    Blog

The File Drawer Problem (Publication Bias)

Publication bias refers to the influence of the results of a study (e.g., whether or not the results are statistically significant, practically significant, or agree with the expectations of the researcher or sponsor) on whether or not the study is published. Publication bias is also called the file drawer problem, especially when the nature of the bias is that studies which fail to reject the null hypothesis (i.e., that do not produce a statistically significant result) are less likely to be published than those that do produce a statistically significant result.

Several studies1 have found evidence of publication bias in the research literature. 

Failing to publish results that are not statistically significant can be particularly problematical. Recall that if a significance level of 0.05 is set, then in repeated studies, about 5% of studies of a situation where the null hypothesis is true will falsely reject the null hypothesis. Thus, if just (or even predominantly) the statistically significant studies are published, the published record mis-represents the true situation. In particular,
The file drawer problem is likely to be even more of a problem when studies have inadequate power. (Example)

Rosenthal2  proposed a method, based on probability calculations, for deciding whether or not a finding is "resistant to the file drawer threat."
Various methods (including "funnel plots"5) have been devised to try to detect publication bias, but may have their own problems.

Recently, clinical trial registries have been instituted in some areas. The hope is that this will help keep track of clinical trials whose results are not published. In particular, certain clinical trials are now required to register at the U. S. National Institutes of
Health site  The director,  Deborah Zarin, was quoted in a 2011 Science article6 as saying,
"We are finding that in some cases, investigators cannot explain their trial, cannot explain their data. Many of them rely on the biostatistician, but some biostatisticians can't explain the trial design.. So there is a disturbing sense of some trials being done with no clear intellectual leader."

Another type of File Drawer problem has been receiving increased attention lately: Data or other information that a published report is based on, but that is not itself published.

References include:

2. R. Rosenthal (1979) The "file drawer problem" and tolerance for null results, Psychological Bulletin, Vol. 86, No. 3, 838-641.

3. J. Scargle (2000) Publication bias: The "file-drawer" problem in scientific inference,  Journal of Scientific Exploration, Vol. 14, No. 1, pp. 91-106.

4. Use of such registries seems to be increasing. See, for example,, where certain clinical trials are now by law required to be registered.

5. See, e.g., Lau, J., et al (2006) The case of the misleading funnel plot, BMJ 333(7568), 597 - 600.

6. Marshall, E. (2011). Unseen world of  clincical trials emerges from US database, Science 333:145.

7. Doshi P., M Jones and T. Jefferson (2012). Rethinking credible evidence synthesis, British Medical Journal 344, Article Number: d7898 DOI: 10.1136/bmj.d7898

Last updated May 13, 2012