COMMON MISTEAKS
MISTAKES IN
USING STATISTICS: Spotting and Avoiding Them
The File Drawer Problem (Publication Bias)
Publication
bias refers to the influence of the results of a study (e.g.,
whether or not the results are statistically significant, practically
significant, or agree with the expectations of the researcher or
sponsor) on
whether or not the study is published. Publication bias is also called
the file drawer
problem,
especially when the nature of the bias is that studies which fail to
reject the null hypothesis (i.e., that do not produce a statistically
significant result) are less likely to be published than those that do
produce a statistically significant result.
Several studies1 have found evidence of
publication
bias in the research literature.
Failing to publish results that are not statistically
significant can
be particularly problematical. Recall that if a significance level of
0.05 is set, then in repeated studies, about 5% of studies of a
situation where the null hypothesis is true will falsely reject the
null hypothesis. Thus, if just (or
even predominantly) the statistically
significant studies are
published, the published record mis-represents the true situation.
In particular,
- Effects that are not real may appear to be
supported by
research.
- Investigators may spend unnecessary effort
conducting
research on topics that have already been well-researched but not
reported.
The file drawer problem is likely to be even more of a
problem when studies have inadequate power. (Example)
Rosenthal2 proposed a method, based on
probability
calculations, for deciding whether or not a finding is "resistant to
the file drawer threat."
- This method has become known as the
fail-safe
file drawer (or FSFD) analysis. It involves calculating a
"fail-safe number" which is used to estimate whether or not the
file-drawer problem is likely to be a problem for a particular review
or meta-analysis.
- However, Scargle3 has
criticized Rosenthal's method on the grounds that it fails to take into
account the bias in the "file drawer" of unpublished studies, and thus
can give misleading results.
- Scargle urges efforts, such as research
registries4, to try to
limit
publication bias. He also suggests that Bayesian methodologies may be
best to deal with the file-drawer problem when combining different
research results in a meta-analysis.
Various methods (including "funnel plots"5) have been
devised to try to detect publication bias, but may have their own
problems.
Recently, clinical trial registries have been instituted in some areas.
The hope is that this will help keep track of clinical trials whose
results are not published. In particular, certain clinical trials are
now required to register at the U. S. National Institutes of Health
site ClinicalTrials.gov.
The director, Deborah Zarin, was quoted in a 2011 Science article6 as
saying,
"We
are finding that in some cases, investigators cannot explain their
trial, cannot explain their data. Many of them rely on the
biostatistician, but some biostatisticians can't explain the trial
design.. So there is a disturbing sense of some trials being done with
no clear intellectual leader."
Another type of File Drawer problem has been receiving increased
attention lately: Data or other information that a published report is
based on, but that is not itself published.
- Increasingly, authors are including such information as on
line supplemental material.
- However, such data from clinical trials can be difficult to
obtain. Doshi et al tried to obtain such material after writing a 2009
Cochrane review on anti-flu medication Tamiflu. Their experience
is described in a recent British Medical Journal article7.
The preface to the article states, ".. the reviewers got access to
thousands of pages of previously unavailable data. [They] describe how
it shook their faith in published reports and changed their approach to
systematic reviews." Their new review based on
the additional data (which was not complete) took the equivalent of 14
months of full-time work by two researchers. Their experience also
points out problems such as lack of standardized definitions.
Notes:
1. References include:
- T. D. Sterling, W. L. Rosenbaum and J. J.
Weinkam (Publication Decisions Revisited: The
Effect of the Outcome of Statistical Tests on the Decision to Publish
and Vice Versa, The American
Statistician, 1995, vol 49 No. 1, pp. 108 - 112) review the
literature through 1995, and report on an additional study indicating
the occurrence of publication bias, with results showing statistical
significance being over-represented than would be expected (although
the
rate depended on the field). They also provide anecdotal evidence that
papers
may be rejected for publication on the basis of having a result that is
not statistically significant.
- F. Song et al (Extent of publication bias
in different
categories of
research cohorts: a meta-analysis of empirical studies, BMC Medical Research Methodology
2009, 9:79, http://www.biomedcentral.com/1471-2288/9/79)
report on a
meta-analysis of studies that examine a cohort of research studies for
publication
bias. In the studies examined, publication bias tended to occur in the
form of
not presenting results at conferences and not submitting them for
publication. This paper also discusses different types of evidence for
publication bias.
- Hopewell, S. et al (Publication Bias
in Clinical
Trials due to Statistical Significance or Direction of Trial Result,
Cochrane Review 2009, Issue 1; abstract available at www.thecochrane library.com)
conclude that "Trials with positive findings are published more often,
and more quickly, than trials with negative findings."
2. R. Rosenthal (1979) The "file drawer problem" and tolerance for null
results, Psychological Bulletin,
Vol. 86, No. 3, 838-641.
3. J. Scargle (2000) Publication bias: The "file-drawer" problem in
scientific inference, Journal
of Scientific Exploration, Vol. 14, No. 1, pp. 91-106.
4. Use of such registries seems to be increasing. See, for example, ClinicalTrials.gov,
where certain clinical trials are now by law required to be registered.
5. See, e.g., Lau,
J., et al (2006) The case of the misleading funnel plot, BMJ 333(7568), 597 - 600.
6. Marshall, E. (2011). Unseen world of clincical trials emerges
from US database, Science
333:145.
7. Doshi P., M Jones and T. Jefferson (2012). Rethinking credible
evidence synthesis, British Medical
Journal 344, Article Number: d7898 DOI: 10.1136/bmj.d7898
Last updated May
13,
2012