Randomization Tests and the Logic of an Experiment

Suppose we want to know if a new teaching technique improves test performance. One approach is to teach one class with the new technique, another with the old technique, and compare the averages.

That is not necessarily an experiment. If the two classes differ in their preparation, instructor, schedule, or anything else consequential, the comparison may tell us as much about the students as it does about the teaching.

We can improve matters by randomly assigning students to the teaching methods. Random assignment does not guarantee identical groups. Instead, it gives us a known mechanism by which the groups differ, allowing us to reason about which differences are surprising.

A deliberately tiny imaginary experiment

Imagine eight participants, randomly assigned so exactly four receive Treatment A and four receive Treatment B. The observed outcomes are:

• A: 6, 7, 8, 9.
• B: 2, 3, 4, 5.

Their sample means are 7.5 and 3.5, so the observed difference is

$$\bar{x}_A-\bar{x}_B=4.$$

Is that large? The answer depends on the question. We need to know what differences would be possible if the treatment had no effect and assignment were the only source of random variation.

The sharp null

A sharp null hypothesis states that each participant’s potential outcome would be unchanged by assignment to either treatment. Under this hypothesis, we can regard all eight observed values as fixed. The labels are what vary.

There are

$$\binom84=70$$

ways to select four of the eight participants for A. If the actual assignment mechanism made all these allocations equally likely, each has probability $1/70$.

We can enumerate the mean difference for every allocation. No central limit theorem or normal approximation is necessary.

For a two-sided alternative, we count assignments producing a difference at least as far from zero as the observed difference of 4. Only two allocations qualify: the four highest outcomes assigned to A, or the four lowest outcomes assigned to A.

Therefore the exact randomization p-value is

$$p_{\text{two-sided}}=\frac{2}{70}\approx0.02857.$$

For a pre-specified one-sided positive alternative, only the highest-to-A allocation qualifies, producing $1/70\approx0.01429$.

The alternative must not be selected after looking at the outcome just because it gives a smaller p-value.

Experiment with all seventy assignments

The laboratory below shows the complete randomization distribution. Switch between the pre-specified two-sided and one-sided comparisons. The underlying results do not change; what counts as extreme does.

Every histogram bar represents the count of assignments yielding a particular mean difference. The accompanying table lists these counts exactly.

Assumptions matter more than arithmetic

This test’s special validity comes from the random assignment design. The p-value assumes that the allocation mechanism was really as described, and that the sharp null is being tested.

If Treatment A was preferentially given to participants with higher baseline performance, the labels would no longer be exchangeable under the same design. Simply rearranging them would not create a valid randomized experiment out of an observational dataset.

We also assume meaningful outcomes, an appropriate definition of the treatment, and no interference between participants under the simple potential-outcomes model. More complicated experiments require methods that respect their actual assignment designs.

Exercise

1. Why are there 70 possible allocations rather than 256?
2. What is the difference between a sharp null and a null statement only about the average treatment effect?
3. If researchers observe the results and then choose the alternative that makes their p-value smallest, what is wrong with the nominal interpretation?

Worked responses

1. We are choosing exactly four participants for group A, with the other four in B: $\binom84=70$.

2. A sharp null fixes every person’s outcome under either assignment and permits the entire randomization distribution to be calculated from the observed values. An average-effect null can hold even when individual effects are not all zero.

3. Choosing the test after observing outcomes changes the analysis procedure and can invalidate the advertised error calibration.

Read further

The ASA GAISE recommendations emphasize technology-enabled investigative reasoning and study design. The OpenStax hypothesis testing chapter gives broader inferential context.

What is fixed and what is randomized

Under a sharp no-effect null, each participant’s outcome would be the same under either treatment. Their observed outcomes therefore stand in for both missing potential outcomes when enumerating the actual allowed assignments. For eight participants with exactly four assigned to A, there are $\binom84=70$ equally likely allocations if that was the design. Outcomes are held fixed and labels are reassigned; this is not random sampling of a new population.

If the observed absolute mean difference is attained or exceeded in two assignments, the exact two-sided p-value is $2/70\approx0.0285714$. No plus-one correction is required when all assignments are enumerated and the observed one is already included. A Monte Carlo subset has different bookkeeping and simulation uncertainty. Ties should count as at least as extreme, consistently with the prespecified statistic.

Blocked randomization permits only assignments respecting the blocks. Cluster randomization moves whole clusters, not individual people. Enumerating all unrestricted individual relabelings in either case would calculate a reference distribution for an experiment that was not conducted. Exchangeability must come from the actual assignment or an explicitly defended model.

The sharp null differs from a zero-average-effect null: individual treatment effects could cancel in the average while remaining nonzero. An exact sharp-null test does not automatically provide an exact test for every weaker null. Its interpretation should retain the null that makes the outcome imputation possible.

Further problems

1. How many allocations choose three treated participants among six?
2. If exactly four of those allocations are at least as extreme, calculate the exact p-value.
3. Why does random assignment among volunteers not establish representativeness of all residents?

Worked further solutions

1. $\binom63=20$.
2. $4/20=0.2$.
3. Assignment supports a treatment comparison within the recruited group. Recruitment determines the relationship of that group to a broader population.