Confidence Intervals and the Meaning of 95 Percent

A newspaper says that a survey result is “accurate to within three percentage points, 95 percent of the time.” That sentence is trying to express uncertainty. Whether it does so accurately depends on how the survey was conducted and what the margin of error means.

Before dealing with complicated real-world surveys, we’ll use a simpler model where the mathematics is exact.

One estimate; many possible samples

Suppose a population has an unknown mean $\mu$. We draw a random sample of size $n$ and calculate the sample mean $\bar X$.

That single number is a point estimate. If we drew another sample, we’d almost certainly get another number.

A confidence interval is constructed by a procedure designed to contain the fixed, unknown population parameter a specified proportion of the time over repeated samples.

The key distinction is between a random interval (before the data are collected) and a fixed parameter.

An idealized example

Suppose individual observations are independent and normally distributed with mean $\mu$ and known population standard deviation $\sigma=10$.

Then the sample mean follows a normal distribution with standard error

$$\mathrm{SE}(\bar X)=\frac{10}{\sqrt n}.$$

The exact 95 percent confidence interval for $\mu$ is

$$\bar X\ \pm\ 1.9599639845\frac{10}{\sqrt n}.$$

If $n=25$ and the observed sample mean is 52, this gives

$$52\pm1.9599639845(2)\approx[48.08,55.92].$$

If we ran the same sampling-and-interval procedure again and again under these assumptions, 95 percent of the resulting intervals would contain $\mu$.

For this idealized model, it is true even for small $n$: normal observations and known $\sigma$ give exact normal sampling behavior.

What 95 percent does not mean

Once the particular interval has been calculated, $\mu$ is still a fixed parameter. In the frequentist interpretation, that interval either contains $\mu$ or it doesn’t.

Saying that there is a 95 percent probability that this particular fixed interval contains this particular fixed parameter is not the usual frequentist interpretation. The 95 percent belongs to the coverage performance of the procedure.

Other frameworks, including Bayesian credible intervals, can assign probability to parameters under explicit probabilistic assumptions. They answer a different question.

Try the coverage experiment

The laboratory below makes the normally hidden repeated-sampling idea visible. It fixes $\mu=50$, draws fifty independent samples, and constructs a separate interval for each.

Change the sample size and nominal level; see how the interval width and misses change. Try repeated batches. Fifty intervals are not many, so the observed fraction may be 92 percent on one run and 98 percent on the next.

The confidence level describes the long-run behavior, not a guarantee about each short batch.

Important real-world caveats

The laboratory knows $\sigma$. Usually, we do not. For inference about a population mean with an unknown standard deviation under appropriate conditions, a Student-$t$ procedure is more typical. For proportions, specialized interval procedures avoid notorious weaknesses of the simplest normal approximation.

No confidence formula automatically repairs selection bias, dependence, bad measurement, or a mispecified model. Coverage claims depend on the data-generating process and the procedure.

Exercise

1. With known $\sigma=12$, $n=36$, and $\bar x=100$, give the approximate 95 percent interval.
2. What changes across repeated confidence-interval experiments: the population mean or the endpoints?
3. Does a 95 percent confidence procedure guarantee that exactly 95 of the first 100 intervals contain the parameter?

Worked answers

1. The standard error is $12/\sqrt{36}=2$; the margin is approximately $1.96(2)=3.92$, giving $[96.08,103.92]$.

2. The sample statistics and interval endpoints vary; the population mean is fixed.

3. No. Coverage is a long-run property, and observed proportions fluctuate in finite collections.

Read further

See OpenStax: Confidence Intervals and the ASA GAISE guidance.

Derive coverage before interpreting one interval

In the independent normal known-$\sigma$ model, $Z=(\bar X-\mu)/(\sigma/\sqrt n)$ is standard normal. The statement $P(-z\leq Z\leq z)=0.95$, with $z=z_{.975}$, can be rearranged as

$$P\left(\bar X-z\sigma/\sqrt n\leq\mu\leq\bar X+z\sigma/\sqrt n\right)=0.95.$$

Before sampling, the endpoints vary. The parameter does not. This derivation is the coverage guarantee for the procedure under its assumptions. After observing one interval, the event that it covers is either true or false, though unknown to us. A Bayesian interval can assign posterior probability under a specified prior and likelihood; it is a different construction, not an alternative verbal gloss for the same frequentist calculation.

For $\sigma=12$ and $n=36$, the standard error is 2. An observed mean of 50 gives interval $50\pm1.959964(2)$, approximately $[46.0801,53.9199]$. It concerns the mean, not 95% of individual observations. To plan margin of error at most $E$ under this model, solve $n\geq(z\sigma/E)^2$ and round up. For $E=3$, the result is $n\geq61.4656$, so use at least 62 independent observations before allowing for missingness.

Further problems

1. How does a 99% interval compare with a 95% interval from the same data and model?
2. Why does quadrupling sample size halve this interval’s width?
3. Explain why the laboratory’s 50 realized intervals need not contain exactly 48 successes at a nominal 95% level.

Worked further solutions

1. The larger critical value gives a wider interval. Higher coverage requires a larger repeated-sampling acceptance region.
2. Width is $2z\sigma/\sqrt n$, so multiplying $n$ by four divides it by two.
3. The number covering is random, with expectation 47.5 under exact calibration and independent repetitions. An expectation is not a required realized count.