Sampling, Bias, and Uncertainty

Imagine a town with 100,000 residents. We want to know what fraction supports a new park. We cannot ask everyone, so we select a thousand people.

An intelligent question is not just how many people did we ask? It is how did those people end up in our sample?

Random sampling and the big idea

In a simple random sample of size $n$, every possible group of $n$ members of the population has an equal chance of being selected.

This does not guarantee that our sample is perfectly representative. By chance alone, it could contain more supporters than opponents. But it gives us a defined random mechanism with which to reason about uncertainty.

If we repeatedly draw random samples, their statistics will differ. This is sampling variability. It remains even when the selection process is impeccable.

Bias is a different problem

Now imagine taking our survey exclusively outside a meeting of people campaigning for the park. Even if we survey ten thousand attendees, we should not expect their answers to represent the town.

That is selection bias. It is not cured simply by collecting more of the wrong observations.

Another concern is nonresponse bias. Even if we select residents randomly, people who do not respond may differ systematically from those who do.

Bias and random error are different concepts. Random sampling error tends to diminish with larger samples, under appropriate conditions. Systematic bias need not.

Try the accompanying response-selection laboratory. The town’s opinions stay fixed. Only the relative probability that two groups respond changes, shifting the expected survey percentage without changing the population percentage. The display reports mathematical expectations, not a simulated or real opinion poll.

Why the square root keeps appearing

For independent, identically distributed observations with finite population variance $\sigma^2$, the sample mean has standard error

$$\mathrm{SE}(\bar X)=\frac{\sigma}{\sqrt{n}}.$$

Doubling a sample does not halve its standard error. To halve the standard error, we generally need four times as many observations.

That relationship is crucial to understanding surveys, clinical studies, and other evidence-based claims. There can be enormous costs associated with modest improvements in precision.

Real survey designs often require adjustments for clustering, weighting, finite populations, and other departures from the ideal model.

Correlation is not randomization

Random sampling helps us learn about a population. Random assignment in an experiment helps us compare interventions by breaking systematic relationships between treatment assignment and potential outcomes.

These ideas are related, but they solve different problems. We can have an experiment with random assignment and a convenience sample, or a representative survey with no experimental intervention.

Check your understanding

1. What happens to the standard error of a sample mean if $n$ increases from 100 to 400?
2. Can an enormous convenience sample have a very small calculated standard error and still be seriously biased?
3. What is the difference between randomly selecting study participants and randomly assigning treatments?

Suggested responses

1. It is halved under the stated model. 2. Yes; the formula for random variability does not eliminate systematic selection bias. 3. The first concerns generalizing to a population, the second concerns making fair comparisons between treatments.

Separate variance, bias, and mean squared error

For an estimator $T$ of a fixed parameter $\theta$, its bias is $E[T]-\theta$ under a specified repetition model. Expanding the square around $E[T]$ gives

$$E[(T-\theta)^2]=\operatorname{Var}(T)+\operatorname{Bias}(T)^2.$$

Larger independent samples can reduce variance while leaving a recruitment bias untouched. A precise estimate of the wrong target can therefore have substantial mean squared error. “Unbiased” is a repeated-procedure property, not a statement that every realized sample estimate is correct.

For simple random sampling without replacement from a finite population of size $N$, an estimated mean’s variance includes a finite-population correction. Using population variance defined with denominator $N-1$, it is $(1-n/N)S^2/n$. The corresponding standard-error factor is $\sqrt{1-n/N}$. Sampling nearly everyone leaves little sampling uncertainty, although measurement error and nonresponse can remain. The familiar independent-with-replacement expression is a useful approximation when the sampling fraction is small.

Additional practice

1. Estimator A has variance 4 and bias 3; estimator B has variance 10 and zero bias. Compare mean squared errors.
2. Under an independent mean model, how much must sample size increase to reduce standard error by a factor of five?
3. If half a finite population is sampled without replacement, what is the standard-error correction relative to the corresponding with-replacement formula using $S^2$?

Worked additional solutions

1. A has MSE $4+9=13$, B has MSE 10. Smaller variance alone does not determine better squared-error performance.
2. Multiply sample size by 25.
3. Multiply by $\sqrt{1/2}\approx0.7071$, under the simple-random-sampling assumptions.