Simpson's Paradox and the Missing Variable

Here is a lovely little statistical nightmare: two treatment methods can show one ranking when we compare all patients, then the opposite ranking within every relevant group.

It is called Simpson’s paradox, though nothing in the arithmetic is actually contradictory. The trick is in what we have left out of the comparison.

A published historical example

In a 1986 British Medical Journal study, researchers compared historical outcomes of kidney-stone treatment methods. One method involved open surgery; another was percutaneous nephrolithotomy.

We will work with the recorded numerical outcomes as a statistical case study, not as clinical guidance. The procedures were compared across different periods and were not randomized into the same patient population.

First look only at the pooled data:

Method	Successful cases	Total	Recorded success proportion
Open surgery (A)	273	350	78.0%
Percutaneous procedure (B)	289	350	82.6%

The pooled percentage is higher for B. If we stopped here, we might conclude that B had better outcomes in the comparison. But we would be leaving out an important variable: stone size.

Now split the data

The published counts can be separated by stone size:

Stone size	A successful / total	B successful / total
Smaller than 2 cm	81 / 87 = 93.1%	234 / 270 = 86.7%
At least 2 cm	192 / 263 = 73.0%	55 / 80 = 68.8%

Within each stone-size category, A has a higher recorded success proportion. How can that be true if B has the higher overall proportion?

Because the two methods were used for different mixtures of cases.

Of the 350 open-surgery observations, 263 involved larger stones. Of the 350 percutaneous observations, only 80 involved larger stones.

Larger-stone cases had lower success proportions under both procedures. The pooled comparisons are therefore weighted very differently.

Weighted averages can reverse an apparent relationship

For method A, the pooled observed success proportion is

$$\frac{87}{350}\frac{81}{87}+\frac{263}{350}\frac{192}{263} =\frac{273}{350}=.780.$$

For B:

$$\frac{270}{350}\frac{234}{270}+\frac{80}{350}\frac{55}{80} =\frac{289}{350}\approx.826.$$

Each overall rate is a weighted average of the within-group rates. The weights are different for the two methods.

When the weights shift enough, the pooled ordering can reverse even though the within-group ordering stays the same.

Play with the comparison

Switch the laboratory from all stone sizes to each size category.

The data do not change. The denominator changes because we are asking a different question about the same dataset.

A single pooled percentage is sometimes a dangerously incomplete summary. Even a correctly calculated statistic can support a misleading story when a relevant variable is omitted.

Does this prove A was the better treatment?

No. These were observational historical comparisons of patients treated under different circumstances and periods. Stratifying by stone size identifies one important imbalance; it does not rule out all other differences, such as patient characteristics, clinician choices, practice changes, or severity within a size category.

Understanding Simpson’s paradox is therefore not a shortcut to causal inference. It is a reason to take study design, context, and multivariable thinking seriously.

The broad lesson extends far beyond medicine. Admissions rates, school achievement, transportation safety, and workplace metrics can all change meaning when we pool distinct groups.

Exercise

1. What fraction of open-surgery cases involved larger stones? What fraction of percutaneous cases?
2. Which method has the higher observed rate in each size group? Which has the higher pooled rate?
3. Why does this dataset, by itself, not prove a causal treatment advantage?

Worked answers

1. $263/350\approx75.1\%$ of open-surgery cases and $80/350\approx22.9\%$ of percutaneous cases involved larger stones.

2. Open surgery has the higher recorded success proportion in both size groups, while the percutaneous method has the higher recorded pooled proportion.

3. The procedures were not randomly assigned to otherwise comparable patients and were observed in different time periods; stone size is one relevant difference, not necessarily the only one.

Data provenance and further reading

The counts are historical results from Charig and colleagues, “Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy,” BMJ (1986), 292:879–882, doi:10.1136/bmj.292.6524.879.

See the Harvard Statistics explanation and a later review of Simpson’s paradox in clinical research. Only published aggregate counts are used; no patient-level records are distributed.

The arithmetic of aggregation

An aggregate success rate is a weighted average of stratum-specific rates: $p_A=\sum_sw_{As}p_{As}$, with weights equal to A’s stratum proportions. B generally has different weights $w_{Bs}$. Consequently, even if $p_{As}>p_{Bs}$ within every stratum, the aggregate inequality need not hold. The comparison mixes within-stratum outcomes with different population compositions.

To isolate composition, a descriptive standardization can use common prespecified weights $w_s$ and compare $\sum_sw_sp_{As}$ with $\sum_sw_sp_{Bs}$. If A is better in every stratum, these common-weight averages preserve that direction. The result describes the chosen standardized population, whose weights must be explained. It is not automatically an estimate of a causal effect: treatment selection and unmeasured prognostic differences may remain within strata.

For a synthetic illustration, A has rates 0.9 and 0.6, B rates 0.8 and 0.5. With common half-and-half weights, standardized rates are 0.75 and 0.65. But if A’s easier-case share is 0.1 and B’s is 0.9, their observed mixture rates are 0.63 and 0.77, reversing the order. These synthetic numbers illustrate the mechanism separately from the cited historical dataset.

Further problems

1. Recalculate the standardized comparison using weights 0.2 and 0.8.
2. Why is the crude aggregate not arithmetically incorrect?
3. Why might conditioning on the wrong variable introduce rather than remove bias?

Worked further solutions

1. A is $0.2(0.9)+0.8(0.6)=0.66$; B is $0.2(0.8)+0.8(0.5)=0.56$.
2. It correctly describes each group’s actual mixture. It answers a different descriptive question from a common-composition comparison.
3. Conditioning on a mediator changes the effect target; conditioning on a common consequence can create an association. Adjustment requires a causal argument about the variable’s role, not a rule to include everything measured.