The File Drawer Problem

COMMON MISTEAKS MISTAKES IN USING STATISTICS: Spotting and Avoiding Them

Introduction Types of Mistakes Suggestions Resources Table of Contents About Glossary

The File Drawer Problem (Publication Bias)

Publication bias refers to the influence of the results of a study (e.g., whether or not the results are statistically significant, practically significant, or agree with the expectations of the researcher or sponsor) on whether or not the study is published. Publication bias is also called the file drawer problem, especially when the nature of the bias is that studies which fail to reject the null hypothesis (i.e., that do not produce a statistically significant result) are less likely to be published than those that do produce a statistically significant result.

Several studies¹ have found evidence of publication bias in the research literature.

Failing to publish results that are not statistically significant can be particularly problematical. Recall that if a significance level of 0.05 is set, then in repeated studies, about 5% of studies of a situation where the null hypothesis is true will falsely reject the null hypothesis. Thus, if just (or even predominantly) the statistically significant studies are published, the published record mis-represents the true situation. In particular,

Effects that are not real may appear to be supported by research.

Investigators may spend unnecessary effort conducting research on topics that have already been well-researched but not reported.

The file drawer problem is likely to be even more of a problem when studies have inadequate power. (Example)

Rosenthal²proposed a method, based on probability calculations, for deciding whether or not a finding is "resistant to the file drawer threat."

This method has become known as the fail-safe file drawer (or FSFD) analysis. It involves calculating a "fail-safe number" which is used to estimate whether or not the file-drawer problem is likely to be a problem for a particular review or meta-analysis.

However, Scargle³has criticized Rosenthal's method on the grounds that it fails to take into account the bias in the "file drawer" of unpublished studies, and thus can give misleading results.

Scargle urges efforts, such as research registries^4_,to try to limit publication bias. He also suggests that Bayesian methodologies may be best to deal with the file-drawer problem when combining different research results in a meta-analysis.

Various methods (including "funnel plots"⁵) have been devised to try to detect publication bias, but may have their own problems.

Notes:
1. References include:

T. D. Sterling, W. L. Rosenbaum and J. J. Weinkam (Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa, The American Statistician, 1995, vol 49 No. 1, pp. 108 - 112) review the literature through 1995, and report on an additional study indicating the occurrence of publication bias, with results showing statistical significance being over-represented than would be expected (although the rate depended on the field). They also provide anecdotal evidence that papers may be rejected for publication on the basis of having a result that is not statistically significant.

F. Song et al (Extent of publication bias in different categories of research cohorts: a meta-analysis of empirical studies, BMC Medical Research Methodology 2009, 9:79, http://www.biomedcentral.com/1471-2288/9/79) report on a meta-analysis of studies that examine a cohort of research studies for publication bias. In the studies examined, publication bias tended to occur in the form of not presenting results at conferences and not submitting them for publication. This paper also discusses different types of evidence for publication bias.

Hopewell, S. et al (Publication Bias in Clinical Trials due to Statistical Significance or Direction of Trial Result, Cochrane Review 2009, Issue 1; abstract available at www.thecochrane library.com) conclude that "Trials with positive findings are published more often, and more quickly, than trials with negative findings."

2. R. Rosenthal (1979) The "file drawer problem" and tolerance for null results, Psychological Bulletin, Vol. 86, No. 3, 838-641.

3. J. Scargle (2000) Publication bias: The "file-drawer" problem in scientific inference, Journal of Scientific Exploration, Vol. 14, No. 1, pp. 91-106.

4. Use of such registries seems to be increasing. See, for example, ClinicalTrials.gov, where certain clinical trials are now by law required to be registered.

5. See, e.g., Lau, J., et al (2006) The case of the misleading funnel plot, BMJ 333(7568), 597 - 600.

Last updated May 12, 2011