"With
careful and prolonged planning, we
may reduce or eliminate many potential sources of bias, but seldom will
we be able to eliminate all of them. Accept bias as inevitable and then
endeavor to recognize and report all exceptions that do slip thought
the cracks."
Good and Hardin (2006) Common
Errors in Statistics (and How to Avoid Them), p. 113
"Unlike
error related to random variability, bias cannot be assessed without
external knowledge of the world"
Herbert I. Weisberg (2010), Bias and Causation: Models and Judgment
for Valid Comparisons, p. 26
A sampling method is called biasedif it systematically favors some outcomes
over others. Sampling bias is sometimes called ascertainment bias (especially in
biological fields) or systematic bias.
Bias can be intentional, but often it is not. The following example
shows how a sample can be biased,
even though there is some
randomness in the selection of the sample.
Example:
Telephone sampling is common in marketing surveys. A simple random
sample may be chosen from the sampling frame
consisting of a list of telephone numbers of people in the area being
surveyed. This method does involve taking a simple random sample, but
it is not a simple random
sample of the target population
(consumers in the area being surveyed.) It will miss people who do not
have a phone. It may also miss people who only have a cell phone that
has an area code not in the region being surveyed. It will also miss
people who do not wish to be surveyed, including those who monitor
calls on an answering machine and don't answer those from telephone
surveyors. Thus the method
systematically excludes certain types of
consumers in the area.
Inferences from a biased sample are not as trustworthy as conclusions
from a truly random sample.
Here are some common sources and consequences of bias:
Convenience samples:
"Statistical
inference with convenience samples is a risky business."
David
A. Freedman, Statistical Models
and Causal Inference, p. 23
Sometimes it is not possible or not practical to choose a random
sample. In those cases, a convenience sample
might be used. Sometimes it is plausible that a convenience sample
could be considered as a random sample, but often a convenience sample
is biased. If a convenience sample
is used, inferences are not
as trustworthy as if a random sample is used.
Voluntary response samples: If
the researcher appeals to people to voluntarily participate in a
survey, the resulting sample is called a "voluntary response sample."
Voluntary response samples are always
biased: they only include people
who choose volunteer, whereas a random sample would need to include
people whether or not they choose to volunteer. Often, voluntary
response samples oversample people who have strong opinions and
undersample people who don't care much about the topic of the survey.
Thus inferences from a voluntary
response sample are not as trustworthy as conclusions based on a random
sample of the entire population under consideration.
Lack of Blinding: When
two "treatments" are compared (e.g., drugs; surgical procedures;
teaching method), bias can sometimes be introduced by the human beings
involved, despite their best efforts to be objective and unbiased. Thus
it is important in these situations to try to make sure that no one who
might, even unintentionally, influence the results knows which
treatment each subject is receiving. This is called blinding.
Examples:
1. If two drugs are being compared (or a drug
and a placebo), blinding involves the following (and possibly more1):
The two pills need to look alike, so the
patient and
the attending medical personnel don't know which drug the patient is
taking.
The person arranging the randomization
(i.e., which
patient takes which drug) should have no other involvement in the
study, and should not reveal to anyone involved in the study which
patient is taking which drug.
Anyone evaluating patient outcomes (e.g.,
examining the
patient or asking the patient about their symptoms) should not know
which drug the patient is taking.
2. Now suppose that two surgical
treatments are being compared. It is still possible to arrange
that the second and third conditions in Example 1 are met, but it is impossible to prevent the
surgeons from knowing which surgical treatment they are giving. Thus,
total blinding is not possible, and there is the possibility that the
surgeon's knowledge of which treatment is being given might influence
the outcome. Sometimes the researchers can partially get around this by
using only surgeons who genuinely believe that the technique they are
using is the better of the two. But this then introduces a confounding
of technique and surgeon: it might be, for example, that the surgeons
preferring one technique are more skilled or more experienced or more
careful than the surgeons preferring the other, or have different
training that affects the outcome regardless of the surgical method.
Miscellaneous Examples:
Studies of human genetic variation typically use DNA
microchips to identify variation in certain genes that are known to
have different verwions. But if the microchip is created to assess only
certain genes known to vary in a particular population, the study
willnot pick up genes that do not vary in that population, but vary
between taht population and others, or within some othe populations.
For example, a study using a microchip based on genes known to vary in
Europran populations may miss variaion betwen European and Asian
populations or between different Asian populations.2
Efron3 describes filtration as "the data-based
preselection of a subset of promising-looking cases for final
analysis." For example, a researcher looking for genes involved in a
certain disease might restrict further analysis to only those genes
that give high standard deviation, considering these as "promising".
But this then affects further analysis, since genes are only being
compared with a subset of all genes originally tested, rather than with
the entire collection.
Extrapolation:
In statistics, drawing a conclusion about something beyond the range of
the data is called extrapolation.
Drawing a conclusion from a biased sample is one form of extrapolation:
because the sampling method systematically excludes certain parts of
the population under consideration, the inferences only apply to the
subpopulation which has actually been sampled. Extrapolation also
occurs if, for example, an inference based on a sample of university
undergraduates is applied to older adults or to adults with only an
eighth grade education. Extrapolation
is a common error in applying
or interpreting statistics.
Sometimes, because of the difficulty or impossibility of obtaining good
data, extrapolation is the best we can do, but it always needs to be
taken with at least a grain of salt -- and often with a large dose of
uncertainty. Notes:
1. For example, if one drug has side effects and the other does not,
the patient or medical personnel might be able to tell from the
incidence of side effects which drug the patient is taking.
2. Box 1, p. 600 in MA Jobling and C Tyler-Smith (2003). The human Y
Chromosome: an evolutionary marker comes of age, Nature Review
Genetics, 4(8): 598 - 612.
3. B. Efron (2010), Large-Scale
Inference: Empirical Bayes Methods for Estimation, Testing, and
Prediction, Cambridge, p. 109