USING STATISTICS: Spotting and Avoiding Them
Suggestions for Readers of Research
experts think peer review validates published research. For those of us
who have been editors, associate editors, reviewers, or the targets of
peer review, this argument may ring hollow. Even for careful readers
of journal articles, the argument may seem a little farfetched."
David A. Freedman, Chance
2008, v. 21 No. 1, p. 61
scientists understand that peer
review per se provides only a minimal assurance of quality, and that
the public conception of peer review as a stamp of authentication is
far from the truth.
Guideline: Look for sources of uncertainty.
- Do not just read the abstract.
- Abstracts sometimes focus on conclusions
more speculative than the data warrant.
- Identify the exact research question(s) the
- Decide if this is a questions that you
- Identify the measures the researchers are
example, if you are interested in the effect of a medication on the
incidence of hip fractures, is this the endpoint that the researchers
have studied, or have they just studied a proxy such as bone density?
- Determine the type of study: observational
experimental; exploratory of confirmatory
- This will influence the strength of the
that can be drawn; generally speaking, experimental studies give
stronger evidence than observational studies, and confirmatory studies
give stronger evidence than exploratory studies.
- Pay attention to how the sample(s) was/were
- Think about any circumstances that might
- Results from a biased sample are
although sometimes they might give some information about a smaller
population than intended.
- Remember that voluntary response samples
- Have the researchers explained why the
procedures they have used are appropriate for the data they are
- In particular, have they given good
reasons why the model assumptions
context well enough?
- If not, their results should be given
if the model has been shown to fit the context well.
- If there is multiple
inference using the same data, have the authors taken that into
account appropriately in deciding significance or confidence levels?
- If hypothesis tests are used, are
- Confidence intervals can give an idea of
the range of
uncertainty due to sampling variability.
- But be aware that there might also be
of uncertainty not captured by confidence intervals (e.g., bias, lack
of fit of model assumptions, measurement uncertainty.)
- Have claims been limited to the population
the data were actually gathered?
- Have the authors taken practical
significance as well
as statistical significance into account in drawing conclusions?
- Is the power of
tests large enough to warrant claims of no difference?
- See Good and Hardin (2010, Chapter 9) for
suggestions and details.
- See van Belle (2008, Chapter 7) for items
Evidence Based Medicine
Good, P. I. and Hardin, J. W, (2010). Common Errors in Statistics, 3rd
van Belle, Gerald (2008). Statistical Rules of Thumb, 2nd ed., Wiley