Introduction        Types of Mistakes        Suggestions        Resources        Table of Contents         About

Biased Sampling and Extrapolation

"With careful and prolonged planning, we may reduce or eliminate many potential sources of bias, but seldom will we be able to eliminate all of them. Accept bias as inevitable and then endeavor to recognize and report all exceptions that do slip thought the cracks."
Good and Hardin (2006) Common Errors in Statistics (and How to Avoid Them), p. 113

"Unlike error related to random variability, bias cannot be assessed without external knowledge of the world"
Herbert I. Weisberg (2010), Bias and Causation: Models and Judgment for Valid Comparisons, p. 26

A sampling method is called biased if it systematically favors some outcomes over others. Sampling bias is sometimes called ascertainment bias (especially in biological fields) or systematic bias.

Bias can be intentional, but often it is not. The following example shows how a sample can be biased, even though there is some randomness in the selection of the sample.


Telephone sampling is common in marketing surveys. A simple random sample may be chosen from the sampling frame consisting of a list of telephone numbers of people in the area being surveyed. This method does involve taking a simple random sample, but it is not a simple random sample of the target population (consumers in the area being surveyed.) It will miss people who do not have a phone. It may also miss people who only have a cell phone that has an area code not in the region being surveyed. It will also miss people who do not wish to be surveyed, including those who monitor calls on an answering machine and don't answer those from telephone surveyors. Thus the method systematically excludes certain types of consumers in the area.

Inferences from a biased sample are not as trustworthy as conclusions from a truly random sample.

Here are some common sources and consequences of bias:

Convenience samples
"Statistical inference with convenience samples is a risky business."
    David A. Freedman, Statistical Models and Causal Inference, p. 23
Sometimes it is not possible or not practical to choose a random sample. In those cases, a convenience sample might be used. Sometimes it is plausible that a convenience sample could be considered as a random sample, but often a convenience sample is biased. If a convenience sample is used, inferences are not as trustworthy as if a random sample is used.

Voluntary response samples: If the researcher appeals to people to voluntarily participate in a survey, the resulting sample is called a "voluntary response sample." Voluntary response samples are always biased: they only include people who choose volunteer, whereas a random sample would need to include people whether or not they choose to volunteer. Often, voluntary response samples oversample people who have strong opinions and undersample people who don't care much about the topic of the survey. Thus inferences from a voluntary response sample are not as trustworthy as conclusions based on a random sample of the entire population under consideration.

Lack of Blinding: When two "treatments" are compared (e.g., drugs; surgical procedures; teaching method), bias can sometimes be introduced by the human beings involved, despite their best efforts to be objective and unbiased. Thus it is important in these situations to try to make sure that no one who might, even unintentionally, influence the results knows which treatment each subject is receiving. This is called blinding.
1. If two  drugs are being compared (or a drug and a placebo), blinding involves the following (and possibly more1):
2. Now suppose that two surgical treatments are being compared. It is still possible to arrange that the second and third conditions in Example 1 are met, but it is impossible to prevent the surgeons from knowing which surgical treatment they are giving. Thus, total blinding is not possible, and there is the possibility that the surgeon's knowledge of which treatment is being given might influence the outcome. Sometimes the researchers can partially get around this by using only surgeons who genuinely believe that the technique they are using is the better of the two. But this then introduces a confounding of technique and surgeon: it might be, for example, that the surgeons preferring one technique are more skilled or more experienced or more careful than the surgeons preferring the other, or have different training that affects the outcome regardless of the surgical method.

Miscellaneous Examples

  1. Studies of human genetic variation typically use DNA microchips to identify variation in certain genes that are known to have different verwions. But if the microchip is created to assess only certain genes known to vary in a particular population, the study willnot pick up genes that do not vary in that population, but vary between taht population and others, or within some othe populations. For example, a study using a microchip based on genes known to vary in Europran populations may miss variaion betwen European and Asian populations or between different Asian populations.2 
  2. Efron3 describes filtration as "the data-based preselection of a subset of promising-looking cases for final analysis." For example, a researcher looking for genes involved in a certain disease might restrict further analysis to only those genes that give high standard deviation, considering these as "promising". But this then affects further analysis, since genes are only being compared with a subset of all genes originally tested, rather than with the entire collection.

Extrapolation: In statistics, drawing a conclusion about something beyond the range of the data is called extrapolation. Drawing a conclusion from a biased sample is one form of extrapolation: because the sampling method systematically excludes certain parts of the population under consideration, the inferences only apply to the subpopulation which has actually been sampled. Extrapolation also occurs if, for example, an inference based on a sample of university undergraduates is applied to older adults or to adults with only an eighth grade education. Extrapolation is a common error in applying or interpreting statistics. Sometimes, because of the difficulty or impossibility of obtaining good data, extrapolation is the best we can do, but it always needs to be taken with at least a grain of salt -- and often with a large dose of uncertainty.

1. For example, if one drug has side effects and the other does not, the patient or medical personnel might be able to tell from the incidence of side effects which drug the patient is taking.
2. Box 1, p. 600 in MA Jobling and C Tyler-Smith (2003). The human Y Chromosome: an evolutionary marker comes of age, Nature Review Genetics, 4(8): 598 - 612.
B. Efron (2010), Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction, Cambridge, p. 109

Updated August 28, 2012