Opening
Imagine testing twenty unrelated hypotheses, reporting the smallest p-value, and neglecting to mention the other nineteen. You have not necessarily made a discovery. You may have simply discovered the minimum function.
Teaching narrative
Type I error concerns rejection under a true null. Type II error and power refer to a specific false null alternative and the procedure's probability of identifying it. Even correct individual p-values can support a misleading conclusion when the family of comparisons is hidden.
A pre-specified test can be calibrated for a five-percent false-rejection rate. But examining many tests and selecting a striking result increases the chance of finding something unusual even when the nulls are true.
Bonferroni adjusts a familywise decision threshold by dividing the desired error level by the number of tests. False-discovery-rate methods aim at another target: the expected proportion of false discoveries among rejected hypotheses under specified conditions.
Exploration is valuable. The wrongdoing is presenting exploratory selections as though they were independent confirmatory tests. Pre-registration, validation data, full reporting, and honest effect sizes make the work more useful.
Before the experiment
What is the chance of at least one false rejection in twenty independent true-null tests, each at alpha=.05?
Live demonstration
Compute 1−.95^20 in R, then apply p.adjust with Bonferroni and Benjamini–Hochberg to a small set of illustrative p-values.
Exit question
Explain the difference between an exploratory result and a confirmatory claim.