What Sort of Study Are We Actually Doing?

Someone discovers that people who drink tea tend to live longer. Should we issue everyone a kettle? Maybe. But the observation itself does not tell us whether tea made anyone live longer.

Before touching a statistical test, we need to understand how the data came into existence.

Observing is not intervening

In an observational study, researchers record exposures, characteristics, and outcomes without assigning the exposure of interest. A survey of commuting habits is observational. So is a study comparing medical outcomes among patients who independently chose different treatments.

A randomized experiment uses a chance mechanism to assign an intervention, such as treatment A or treatment B. Under a suitable design, randomization prevents investigators and participants from systematically choosing treatment groups based on pre-treatment characteristics.

Random assignment does not mean that two realized groups will have precisely the same age, income, or health. Chance still produces imbalances. Rather, those imbalances arise from a known mechanism we can use in inference.

Random sampling is a different machine

Imagine recruiting 100 volunteers and randomly assigning 50 to each intervention. The experiment may support causal inference about the volunteer population under its assumptions, but it does not automatically represent everyone in a city.

Random sampling concerns who enters the study and the population to which we can generalize. Random assignment concerns why groups differ on treatment. They serve distinct purposes.

Controls, blinding, and ethics

A control group supplies a meaningful comparator: usual practice, placebo, or another intervention. A placebo may help distinguish treatment effects from expectations, but it is not ethically acceptable to deny established effective care merely to make a cleaner experiment.

Blinding can reduce measurement or behavior changes related to knowledge of treatment assignments. When blinding is impossible, clearly measured endpoints, pre-registered outcomes, and independent assessment can still improve the study.

In medical and social research, informed consent, safety monitoring, privacy, and ethical oversight are parts of the design—not paperwork to bolt on after the results are interesting.

A small example

A school compares 20 students who enrolled in an optional tutoring program with 20 who did not. The tutoring group gains four more points on average. Maybe tutoring helped. Maybe students seeking tutoring were more motivated or had different prior preparation. A difference between means does not resolve this.

If the school randomly offered access to tutoring, that can improve causal identification of the offer’s effect. It may not identify the effect of actually attending tutoring when attendance differs from the assignment.

Check yourself

1. A random telephone survey asks about voting intentions. Is treatment randomly assigned?
2. A randomized trial recruits only university students. What population-generalization concern remains?
3. Why might measuring outcomes without knowing treatment assignment improve an experiment?

Worked responses

1. No. Random selection for observation and treatment assignment are different operations.

2. University student volunteers may not represent the broader population.

3. Blinding can reduce biased outcome measurement or interpretation linked to treatment knowledge.

Further reading

See the experimental-design chapter of OpenStax Introductory Statistics 2e.

A model in which sampling and assignment can be separated

Consider a hypothetical university of 1,000 students. Half enter with more prior preparation, and half have less. Suppose a workshop would help both groups, but its benefit is larger for those with less preparation.

Imagine that 90% of the more-prepared students volunteer for a study but only 20% of the less-prepared students do. The volunteers are not a miniature copy of the university. Because effects differ between groups, the average treatment benefit among volunteers need not equal the benefit in the university population.

Now suppose the researchers randomize workshop access among those volunteers. This improves our ability to learn an effect among the selected participants. It does not turn the volunteers into a representative sample.

Finally, imagine that researchers instead allow students to choose whether to attend. More-prepared students may select into treatment at a different rate. The raw difference between those who attended and those who did not can combine a real treatment effect with differences in prior preparation.

These are distinct mechanisms. Random sampling addresses who we can generalize to. Random assignment addresses a causal comparison. Neither automatically supplies the other.

Work through each mechanism in the interactive study-design studio. The model reports expected counts rather than actual experimental data, so a perfectly balanced expectation is not a promise of perfectly balanced real treatment groups.

Assignment, receipt, and analysis population

Random assignment specifies the offered intervention, but participants may not receive or adhere to it. An intention-to-treat comparison follows assigned groups and estimates the effect of the assignment strategy under its design. Comparing only those who complied can reintroduce selection: people who adhere may differ in prognosis or resources. A treatment-received analysis needs additional assumptions to identify a causal effect.

Suppose 50 volunteers are assigned to each arm of a hypothetical trial. Ten assigned to treatment never attend, and five controls obtain similar services elsewhere. Discarding these 15 people destroys the original randomized comparison unless the missingness or compliance mechanism is addressed. Retaining assignment may dilute the difference relative to perfect adherence, but it preserves a clearly defined policy question about offering the treatment.

Blocking randomizes within groups such as site or baseline severity to improve balance or precision. Blinding reduces some behavior and measurement changes associated with knowledge of assignment. Neither replaces ethical consent or outcome validity. State who was randomized, the allocation unit, the implemented mechanism, and the planned analysis population before interpreting a treatment coefficient.

Further practice

1. Distinguish an allocation unit from an outcome-measurement unit in a school-randomized study.
2. Why can analyzing only adherent participants defeat randomization?
3. Does an observational study become randomized when a computer randomly orders its rows?

Worked further responses

1. Schools receive assignments; outcomes may be measured on pupils nested within them. The analysis must respect the assignment unit and dependence.
2. Adherence is a postassignment behavior potentially associated with outcomes. Selecting on it can make analyzed groups systematically different.
3. No. Row ordering does not change how exposure or treatment was assigned to people.