The Nineteen Tests Nobody Mentioned

Imagine running twenty independent significance tests, each at the 5% threshold, when all twenty null hypotheses are true.

The probability of avoiding a false rejection on one test is $.95$. The probability of avoiding one on all twenty is $.95^{20}$, assuming independence. Therefore

$$P(\text{at least one false rejection})=1-(.95)^{20}\approx .6415.$$

Roughly 64%. This is why trying twenty models and reporting only the most exciting one changes the statistical argument.

The family is part of the study

What constitutes the group or family of tests depends on the substantive question and inferential goals. It may include multiple outcomes, subgroups, predictors, and model specifications.

If a researcher examines twenty outcomes, selectively reports the smallest p-value, and describes it as the result of one planned test, the nominal evidence is overstated.

Independence made the numerical example easy. Tests in a real analysis are often correlated, so the exact false-positive calculation may differ. The basic problem remains.

Bonferroni: a simple guardrail

To control the probability of at least one false rejection across $m$ hypotheses, Bonferroni compares each p-value to $\alpha/m$.

With 20 tests and a familywise target of $.05$,

$$\alpha_\text{individual}=.05/20=.0025.$$

This can be conservative, but it works under very broad dependence conditions when the individual tests’ p-values are valid.

Other procedures, such as Benjamini–Hochberg, can target the false discovery rate, the expected fraction of false discoveries among rejected hypotheses under appropriate conditions. That is a different goal from controlling the chance of any false discovery.

Exploration is good. Pretending it was confirmation isn’t.

Looking for patterns can generate valuable hypotheses. But a pattern discovered because it looked unusually strong should not automatically be presented as independently confirmed.

Pre-registration, a separate validation dataset, clear description of the analysis history, and appropriate adjustments can help. Transparency is more important than a ritual of declaring a p-value significant.

Example

A single pre-planned test produces $p=.03$. Under a predeclared $.05$ rule, it rejects its null, subject to the assumptions.

The same $p=.03$ is discovered after trying twenty tests. It no longer meets a Bonferroni familywise threshold of $.0025$. The data have not changed, but the procedure that selected the reported result has.

Check your understanding

1. What Bonferroni threshold corresponds to ten tests and a familywise level of .05?
2. Why does reporting only the smallest p-value from many tests distort its interpretation?
3. Must all exploratory statistical work be abandoned?

Worked responses

1. $.005$.

2. Selection favors random extremes, invalidating a single-test reading if multiplicity is ignored.

3. No. Exploration is essential; it should be labeled honestly, and confirmatory claims require suitable validation.

Two procedures with different error targets

Bonferroni follows from the union bound: $P(\cup_iE_i)\leq\sum_iP(E_i)$. If each true null has false-rejection probability at most $\alpha/m$, the probability of any false rejection is at most $\alpha$, without assuming independence. Strong dependence can make the bound conservative, but does not invalidate it when individual p-values are valid.

Holm’s step-down procedure sorts the p-values $p_{(1)}\leq\cdots\leq p_{(m)}$. Compare the smallest with $\alpha/m$, the next with $\alpha/(m-1)$, and so on, stopping at the first failure. It controls familywise error under broad dependence conditions and never rejects fewer hypotheses than ordinary Bonferroni on the same family.

Benjamini–Hochberg instead finds the largest $k$ with $p_{(k)}\leq kq/m$ and rejects the first $k$. Its usual false-discovery-rate guarantee holds under independence and certain positive-dependence conditions. It is not a universal arbitrary-dependence guarantee. If there are no rejections, define the false-discovery proportion as zero; FDR is its expectation, not the realized false fraction in every study.

For p-values $[0.001,0.01,0.03,0.2]$ with target $0.05$, Bonferroni rejects the first two because the common threshold is $0.0125$. Holm rejects the first two and stops at $0.03>0.025$. BH compares with $[0.0125,0.025,0.0375,0.05]$, finds $k=3$, and rejects the first three. The extra rejection reflects a different error criterion, not a calculation error.

Further problems

1. Apply BH at $q=0.05$ to $[0.01,0.04,0.08]$.
2. Explain why searching twenty outcomes but adjusting only the two reported ones conceals multiplicity.
3. What must be stated when reporting an adjusted p-value?

Worked further solutions

1. Thresholds are approximately $0.01667,0.03333,0.05$. Only the first qualifies, so reject one.
2. Selection occurred across all twenty; the adjustment must reflect the inferential family and selection procedure, not just the survivors.
3. Name the procedure, family, target error rate, and relevant dependence assumptions, alongside effect estimates and uncertainty.