This site is under construction. Please check back every few weeks for updates

COMMON MISTEAKS MISTAKES IN USING STATISTICS: Spotting and Avoiding Them

Introduction        Types of Mistakes        Suggestions        Resources        Table of Contents         About



Over-Interpretation

Authors and readers alike often succumb to the temptation to over-interpret the results of statistical studies. Some common mistakes of this form include:

Extrapolation to a larger population than the one studied 

Examples:
Extrapolation beyond the range of data

This is similar to extrapolating to a larger population, but concerns the values of the variables rather than the individuals. More

Ignoring Ecological Validity

This is also similar to extrapolating to a larger population, but the extrapolation may involve the setting (i.e., the "ecology") rather than the individuals studied, or it may involve extrapolation to a population having characteristics very different from the population that is relevant for application.

Examples:
1. An experiment designed to study whether an abstract or concrete approach works better for teaching abstract concepts1 used computer-delivered instruction. This was done to avoid confounding variables such as the effect of the teacher. However, the study then lacked ecological validity for most real-life classroom instruction.

2. A review of articles on molecular diagnostic tests2 found that "Of 108 articles included in the study, 82 (76%) used a design that used healthy controls or alternative-diagnosis controls, only 15 (11%) addressed a clinically relevant population similar to that in which the test might be applied in practice." The use of a control group not from a clinically relevant population is an instance of lack of ecological validity.

Using overly strong language in stating results

Statistical procedures do not prove results. They only give us information on whether or not the data support or are consistent with a particular conclusion. There is always uncertainty involved. Thus results need to be phrased in ways that acknowledge this uncertainty.

Considering statistical significance but not practical significance

Example: Suppose that a well-designed, well-carried out, and carefully analyzed study shows that there is a statistically significant difference in life span between people engaging in a certain exercise regime at least five hours a week for at least two years and those not following the exercise regime. If the difference in average life span between the two groups is three days, ... well, so what?


1. J.A. Kaminski, V.M. Sloutsky, & A.F. Heckler. 2008. The Advantage of Learning Abstract Examples in Learning Math. Science. Vol. 320: 454-455. This study also presents other problems in interpretation and applicability. For example, teh subjects were college students, so applying the results to elementary school students would be a serious extrapolation.

2. B. Lumbreras et al (2009),
Overinterpretation of clinical applicability in molecular diagnostic research, Clin Chem. 2009 Apr;55(4):786-94. The quote is from the abstract. The authors also point out other types of overinterpretation.