How We Choose a Sample Changes the Mathematics

Suppose we want to survey a school district with forty schools. It would be convenient to visit two schools, interview everyone there, and call the result a district survey.

That might work for some questions. But the students in a school often resemble one another more than they resemble students at another school. Convenience is not a sampling design.

Simple random and stratified samples

In a simple random sample of size $n$, every set of $n$ population units has the same chance of selection.

In a stratified sample, we divide the population into defined groups (strata) and draw probability samples separately from each. If the strata are meaningful—say, elementary versus high schools—this can ensure each is represented and may improve precision.

However, oversampling one stratum means its raw sample percentage cannot necessarily be used as the overall population percentage. Appropriate design weights may be needed.

Cluster sampling

A cluster sample selects groups, such as schools, then studies all or some units inside the selected groups. This can be much cheaper than visiting hundreds of dispersed locations.

But outcomes within one cluster may be correlated. A sample of 500 students from two schools does not necessarily supply as much independent information as 500 students sampled across the district.

Under common approximations for equally sized clusters, the design effect can be related to cluster size $m$ and intracluster correlation $\rho$:

$$\operatorname{DEFF}\approx1+(m-1)\rho.$$

For $m=25$ and $\rho=.10$, this approximation gives $1+24(.10)=3.4$. The standard error can be roughly $\sqrt{3.4}$ times an independent-sample calculation under the corresponding conditions.

This is a teaching approximation, not a universal survey-analysis formula.

Systematic selection and nonresponse

Selecting every tenth name from an ordered list can be useful when the starting point is randomized. But if the list has periodic structure, systematic sampling can produce unexpected bias.

Even a beautifully designed probability sample faces nonresponse. People may answer at different rates, and missing respondents may differ in the outcome itself. We explored that in the selection bias laboratory.

What a useful survey report must say

Describe the target population, frame, sample design, selection probabilities, response rate, weighting, missingness, and uncertainty method. “Surveyed 5,000 people” does not tell us whether those people are representative.

Check yourself

1. What is one advantage of stratifying by school level?
2. Why might two schools with 500 respondents require a different variance calculation than a simple random sample of 500 students?
3. What is the approximate design effect when $m=11$ and $\rho=.05$ under the simplified formula?

Worked responses

1. It ensures each school level contributes observations and can reduce variability within strata.

2. Students within schools may share characteristics; clustering reduces independent information.

3. $1+(11-1)(.05)=1.5$.

Weights change the target of a sample mean

Suppose a target population is 80% group A and 20% group B, but a stratified sample deliberately selects 50 from each. If observed group means are 10 and 30, the unweighted mean is 20. The population-composition estimate is $0.8(10)+0.2(30)=14$. Both summarize numbers correctly, but they target different mixtures. Report the population weights, their source, and the sampling design.

For independent simple random samples within strata, an estimated variance of $\hat\mu=\sum_hW_h\bar x_h$ is $\sum_hW_h^2(1-n_h/N_h)s_h^2/n_h$, where $W_h=N_h/N$ and the finite-population correction reflects sampling without replacement. Neglecting the correction may be reasonable when fractions are small, but ignoring weights and design altogether is a different and often consequential approximation.

For roughly equal cluster size $m$ and intracluster correlation $\rho$, the approximation $DEFF=1+(m-1)\rho$ compares variance with an independent-sample reference. If 100 observations have design effect 2.5, a rough effective sample size is 40 and the standard-error inflation is $\sqrt{2.5}\approx1.581$. This is a planning approximation, not a universal formula for arbitrary weights, unequal clusters, or all statistics.

Inverse-probability weights require meaningful inclusion probabilities and an appropriate sampling frame. Calibration to known margins cannot guarantee correction for every unmeasured nonresponse mechanism. Extreme weights can increase variance, so a weighted estimate needs an uncertainty calculation that respects its construction.

Further problems

1. A population is 30% A and 70% B, with estimated stratum means 5 and 15. Find the composition-weighted mean.
2. Under the equal-cluster approximation, find design effect for $m=10$, $\rho=0.2$.
3. Why are many pupils from two classrooms not equivalent to the same number from many independently selected classrooms?

Worked further solutions

1. $0.3(5)+0.7(15)=12$.
2. $1+9(0.2)=2.8$.
3. Shared classroom influences create within-cluster dependence. Additional pupils may supply less independent information than additional classrooms.