A Regression Slope Needs a Standard Error

A regression program prints a slope of 0.6. Is that a strong association? It depends on what one unit of the predictor means, how spread out the observations are, and how uncertain the estimate is.

We should not stop at the fitted line.

Where the slope comes from

For $n$ paired observations, let

$$S_{xx}=\sum_i(x_i-\bar x)^2,\quad b_1=\frac{\sum_i(x_i-\bar x)(y_i-\bar y)}{S_{xx}}.$$

Once the intercept is fitted, the residual sum of squares is

$$SSE=\sum_i(y_i-\widehat y_i)^2.$$

Under the usual independent constant-variance normal-error model, estimate the error variance as $s_e^2=SSE/(n-2)$. The standard error of the slope is then

$$\widehat{\operatorname{SE}}(b_1)=\sqrt{\frac{s_e^2}{S_{xx}}}.$$

The $n-2$ denominator reflects estimating both the intercept and slope.

A fully worked example

Consider five fictional paired observations:

x	y
1	2
2	4
3	5
4	4
5	5

Here $\bar x=3$, $\bar y=4$, $S_{xx}=10$, and $S_{xy}=6$. The fitted line is

$$\widehat y=2.2+0.6x.$$

The residuals are $[-.8,.6,1,-.6,-.2]$, so $SSE=2.4$. Thus $s_e^2=2.4/3=.8$, and the slope standard error is $\sqrt{.8/10}\approx.2828$.

For the null $H_0:\beta_1=0$, the t statistic is $0.6/.2828\approx2.121$, with three residual degrees of freedom under the stated model.

The sample is tiny; apparent positive slopes can still carry considerable uncertainty.

What else must be checked?

The usual test relies on a statistical model for observations and errors. Curvature, unequal residual variance, dependence, influential observations, or incorrect functional form can undermine a naive t reference distribution.

For multiple regression, slope coefficients also depend on which other predictors are included. Correlated predictors can increase uncertainty and complicate interpretation.

And none of this alone proves changing x causes y to change.

Reproduce with base R

fit <- lm(c(2,4,5,4,5) ~ c(1,2,3,4,5))
summary(fit)
confint(fit)

Report the coefficient in its original units, its uncertainty, design assumptions, and limitations.

Check your understanding

1. Why is the residual degrees of freedom $n-2$ in a simple intercept-plus-slope model?
2. Does a high $R^2$ guarantee a significant causal slope?
3. Why would extrapolation far beyond observed x values be risky?

Worked responses

1. Two coefficients (intercept and slope) were estimated from the observations.

2. No. Association, uncertainty, and causality are distinct claims.

3. The fitted linear relationship may not hold outside the observed range.

An interval, a mean response, and a new observation

For the five-point example, $SE(b_1)=\sqrt{0.08}\approx0.282843$ and $t_{.975,3}\approx3.182446$. A 95% slope interval is therefore

$$0.6\pm3.182446\sqrt{0.08}\approx[-0.30013,1.50013].$$

The data show a positive fitted slope but also substantial uncertainty under the classical model. This interval refers to change in the conditional population mean response per unit of predictor, not a prediction interval for a new response.

At a specified predictor $x_0$, the estimated conditional mean is $\hat y_0=b_0+b_1x_0$. Its standard error is $s_e\sqrt{1/n+(x_0-\bar x)^2/S_{xx}}$. For a new independent observation at that predictor, the prediction standard error is $s_e\sqrt{1+1/n+(x_0-\bar x)^2/S_{xx}}$. The extra one represents the new observation’s residual variability. Under the normal-error model, matching t critical values produce mean-response and prediction intervals, respectively.

At $x_0=3$ in the example, $\hat y_0=4$, mean-response standard error $\sqrt{0.8/5}=0.4$, and new-observation standard error $\sqrt{0.8(1.2)}\approx0.97980$. The resulting intervals are about $[2.7270,5.2730]$ and $[0.8818,7.1182]$. Prediction is harder than estimating a conditional mean. Both intervals widen away from the observed predictor center, even before we confront the additional risk that the model ceases to hold outside its observed range.

These formulas treat predictor values as fixed or condition on them, assume correctly specified linear conditional means and independent constant-variance errors, and use normal errors for exact small-sample t calibration. Robust standard errors address some variance misspecification but do not repair nonlinear mean structure, selection, or confounding.

Additional practice

1. Holding error variance and predictor spread per observation similar, why does increasing sample size tend to reduce slope uncertainty?
2. Why is a prediction interval wider than a conditional-mean interval at the same input?
3. If all predictor values are identical, which formula fails and what does that mean?

Worked additional solutions

1. $S_{xx}$ generally grows with the number of independent observations, reducing $s_e/\sqrt{S_{xx}}$.
2. A new observation has its own residual variability in addition to coefficient-estimation uncertainty.
3. $S_{xx}=0$, so the slope is not identified. A flat predictor column supplies no observed variation with which to estimate a response trend.