COMMON MISTEAKS MISTAKES IN USING STATISTICS: Spotting and Avoiding Them

Introduction        Types of Mistakes        Suggestions        Resources        Table of Contents         About


Data Snooping

Data snooping refers to statistical inference that the researcher decides to perform after looking at the data (as contrasted with pre-planned inference, which the researcher plans before looking at the data).

Data snooping can be done professionally and ethically, or misleadingly and unethically, or misleadingly out of ignorance.  Data snooping
misleadingly out of ignorance is a common error in using statistics. The problems with data snooping are essentially the problems of multiple inference.

One way in which researchers unintentionally obtain misleading results by data snooping is in failing to account for all of the data snooping they engage in. In particular, in accounting for Type I error when data snooping, you need to count not just the actual hypothesis tests performed, but also all comparisons looked at when deciding which post hoc (i.e., not pre-planned) hypothesis tests to try.

Example:  A group of researchers plans to compare three dosages of a drug in a clinical trial.  There is no pre-planned intent to compare effects broken down by sex, but the sex of the subjects is recorded. The researchers have decided to have an overall Type I error rate of 0.05, allowing 0.03 for the pre-planned inferences and 0.02 for any data snooping they might decide to do. The pre-planned comparison shows no statistically significant difference between the three dosages when the data are not broken down by sex. However, since the researchres have recorded sex of the patients, they decide to look at the outcomes broken down by combination of sex and dosage. They notice that the results for women in the high-dosage group look much better than the results for the men in the low dosage group, and perform a hypothesis test to check that out. In accounting for Type I error, the researchers need to take the number of data-snooping inferences performed as 15, not one. The reason is that they have looked at fifteen comparisons:  there are 3×2 = 6 dosage-by-sex combinations, and hence (6×5)/2 = 15 pairs of dosage-by-sex combinations. Thus the significance level for the post hoc test should not be 0.02. but 0.02/151.

For some discussions of multiple inference and data snooping with a humerous slant, see:

Seife, Charles, The Mind-Reading Salmon: The True Meaning of Statistical Significance, Scientific American, August 21, 2011

XKCD, Significant

Suggestions for data snooping professionally and ethically

I. Educate yourself on the limitations of statistical inference: Model assumptions,  the problems of Types I and II errors, power, and multiple inference, including the "hidden comparisons" that may be involved in data snooping (as in the above example).

 II. Plan your study to take into account the problems involving model assumptions, Type I and II errors, power, multiple inference. Some specifics to consider:

a. If you will be gathering data, decide before gathering the data:
Then do a power analysis to see what sample size is needed to meet these criteria.
 b. If you plan to use existing data, you will need to go through a process similar to that in (a) before looking at the data:
Then do a power analysis to see what sample size is needed to meet these criteria.
c. If data snooping is intended to be the purpose or an important part of your study, then before you look at the data, divide it randomly into two parts: One to be for used for discovery purposes (generating hypotheses), the other to be used for confirmatory purposes (testing hypotheses).
III. Report your results carefully, aiming for honesty and transparency

Notes:
1. This is assuming a Bonferroni procedure. If another multiple inference procedure is available, it might give an effective individual significance level somewhat higher than 0.02/15


This page last revised 11/3/2011