Week 6 · Dr. Aluma Dembo

Hypothesis Tests & p-values

Hypothesis Tests & p-values

Part of: Intro Econometrics | Key concepts: Hypothesis Testing, Null Hypothesis, t-statistic, Critical Value, p-value, Confidence Interval, One-Sided Test, Regression Through the Origin

How this note connects

The slides open with a recap of confidence intervals (Lecture 5 material). This note keeps that recap short and spends its length on the new material: t-tests, p-values, one-sided tests and regression through the origin. For the wider course (OLS, t and F tests, CIs) see Lec_13-Exam Review; for the regression-through-the-origin homework see Assign_03-Change of Variables, Regression Through the Origin & the Housing Model.


1. Recap: confidence intervals (from last time)

  • A 1−α1-\alpha confidence interval satisfies P(L≤T≤U)=1−α\mathbb{P}(L \le T \le U) = 1-\alpha. Here α\alpha is the significance level (usually 5%) and 1−α1-\alpha the confidence level (usually 95%).
  • With the symmetric t-distribution, the critical value tα/2[df]t_{\alpha/2}[df] solves P(T≤tα/2)=1−α/2\mathbb{P}(T \le t_{\alpha/2}) = 1-\alpha/2, so P(−tα/2≤T≤tα/2)=1−α\mathbb{P}(-t_{\alpha/2} \le T \le t_{\alpha/2}) = 1-\alpha. (Slides 3 to 11 build this with CDF shading: P(c1≤T≤c2)=F(c2)−F(c1)\mathbb{P}(c_1\le T\le c_2)=F(c_2)-F(c_1).)
  • Under assumptions A1 to A5 (A5: normal errors ui∼N(0,σ2)u_i \sim \mathcal{N}(0,\sigma^2)), β^1\hat\beta_1 is normal, and replacing the unknown σ2\sigma^2 with σ^2=1N−2∑u^i2\hat\sigma^2 = \frac{1}{N-2}\sum \hat u_i^2 gives
t(β^1)=β^1−β1se(β^1)∼t[N−2]t(\hat\beta_1)=\frac{\hat\beta_1-\beta_1}{se(\hat\beta_1)} \sim t[N-2]
  • Hence the CI:
β^1±tα/2[df]⋅se(β^1)\boxed{\hat\beta_1 \pm t_{\alpha/2}[df]\cdot se(\hat\beta_1)}
Activity 6 (part 1): 95% CI for hours

Wage model ln_earnings ~ s + exp + IQ_test + age + hours, n=540n=540, 534 df, t0.025(∞)=1.96t_{0.025}(\infty)=1.96. For hours: estimate 0.00415050.0041505, s.e. 0.00246870.0024687.

0.00415±1.96×0.00247=0.00415±0.00484=[−0.00069, 0.00899]0.00415 \pm 1.96\times 0.00247 = 0.00415 \pm 0.00484 = [-0.00069,\ 0.00899]. The interval contains 0, which foreshadows hours not being significant at 5%.

Estimation vs inference (slide 18):

Stage Question
Estimation Given NN observations, which β^\hat\beta fits best?
Confidence interval How likely is it that β^\hat\beta falls in a range around the true β\beta?
Hypothesis test (today) How probable is our sample if the true parameter were a specific value bb?

2. The two-sided t-test: four steps

A test never gives a certain "yes/no", only "with some probability no". Four steps:

  1. State H0H_0 and H1H_1. In general H0:β1=bH_0:\beta_1 = b against H1:β1≠bH_1:\beta_1 \ne b. Examples: slope equals 2; intercept equals 4; slope is zero ("does X affect Y?").
  2. Compute the t-statistic, taking β^1\hat\beta_1 and se(β^1)se(\hat\beta_1) from the output and bb from the null:
t(β^1)=β^1−bse(β^1)\boxed{t(\hat\beta_1)=\frac{\hat\beta_1-b}{se(\hat\beta_1)}}
  1. Pick the significance level α\alpha (usually 5%; 10% and 1% are also checked) and find the critical value tα/2[df]t_{\alpha/2}[df] from the t-table.
  2. Decide.
    • If −tα/2≤t(β^1)≤tα/2-t_{\alpha/2}\le t(\hat\beta_1)\le t_{\alpha/2}: fail to reject H0H_0. If H0H_0 is true this happens with probability 1−α1-\alpha.
    • If ∣t(β^1)∣≥tα/2|t(\hat\beta_1)| \ge t_{\alpha/2}: reject H0H_0. If H0H_0 is true this (a false rejection) happens with probability α\alpha, split as α/2\alpha/2 in each tail.

Lec06_rejection_regions Theory diagram: rejection regions (red) and non-rejection region (blue) for a two-sided and a right-tail test at 5%, df = 32.

Wording

Say "fail to reject", never "accept". Not rejecting H0H_0 only means the data are consistent with it; many other values of bb would also survive.

Example 1: income-consumption model

Consumption=β0+β1GDP+u\text{Consumption} = \beta_0+\beta_1\text{GDP}+u (Israeli data, 32 df). Output: β^1=0.81547\hat\beta_1 = 0.81547, se=0.01914se = 0.01914, β^0=−21894.88752\hat\beta_0=-21894.88752, se=1376.81459se=1376.81459. Critical value t0.025[32]=2.0369t_{0.025}[32]=2.0369.

Question (all at α=5%\alpha=5\%) H0H_0 tt Decision
Does income affect consumption? β1=0\beta_1=0 0.81547/0.01914=42.610.81547/0.01914 = 42.61 Reject: income matters
Do Israeli consumers not save? (MPC = 1) β1=1\beta_1=1 (0.81547−1)/0.01914=−9.64(0.81547-1)/0.01914=-9.64 Reject (slide left this blank; reconstructed)
Is the marginal propensity to consume 80%? β1=0.8\beta_1=0.8 (0.81547−0.8)/0.01914=0.808(0.81547-0.8)/0.01914=0.808 Fail to reject

Example 2: housing prices

ln⁡Price=β0+β1ln⁡sqrft+β2 bdrms+u\ln Price = \beta_0+\beta_1\ln sqrft+\beta_2\,bdrms+u, 88 US apartments, 85 df. The slides use t0.025[60]=2.00t_{0.025}[60]=2.00 (nearest table row).

Coefficient Estimate Std. Error t value Pr(>|t|)
(Intercept) 6.28435 0.69758 9.009 5.18e-14
lsqrft 0.80826 0.09869 8.190 2.35e-12
bdrms 0.03811 0.03034 1.256 0.213

R2=0.5611R^2=0.5611, adjusted R2=0.5508R^2=0.5508, F=54.34F=54.34 on 2 and 85 df.

Question H0H_0 tt Decision
Bedrooms do not affect price β2=0\beta_2=0 1.2561.256 Fail to reject
One more bedroom cuts price by 2% β2=−0.02\beta_2=-0.02 slide: 1.9331.933 Fail to reject
One more bedroom raises price by 10% β2=0.1\beta_2=0.1 slide: −2.063-2.063 Reject
Rounding on slides 36 and 37

With se=0.03034se=0.03034 I get t≈1.915t\approx1.915 for H0:β2=−0.02H_0:\beta_2=-0.02 and t≈−2.04t\approx-2.04 for H0:β2=0.1H_0:\beta_2=0.1. The slide values (1.933, -2.063) look like they use se≈0.030se\approx0.030. The decisions are the same either way, and the −2.04-2.04 is only just beyond 2.002.00.


3. Confidence intervals and two-sided tests are the same thing

We fail to reject H0:β1=bH_0:\beta_1=b iff −tα/2≤β^1−bse≤tα/2-t_{\alpha/2}\le \frac{\hat\beta_1-b}{se}\le t_{\alpha/2}, and rearranging (slide 32) gives

β^1−tα/2 se(β^1) ≤ b ≤ β^1+tα/2 se(β^1)\hat\beta_1-t_{\alpha/2}\,se(\hat\beta_1)\ \le\ b\ \le\ \hat\beta_1+t_{\alpha/2}\,se(\hat\beta_1)
Rule

If bb is inside the (1−α)(1-\alpha) CI, fail to reject at α\alpha. If bb is outside, reject. This holds for two-sided tests only.

  • Income-consumption: CI [0.776, 0.854][0.776,\ 0.854] contains 0.8 (not rejected) but excludes 0 and 1 (rejected). Consistent with the three tests above.
  • Housing: CI [−0.0225, 0.0987][-0.0225,\ 0.0987] (slides use ±2×0.0303\pm 2\times0.0303) contains −0.02-0.02 and 00 but excludes 0.10.1. Again consistent.

4. Significance testing and p-values

Significance test

"Does xkx_k have a significant effect on yy?" is the special case H0:βk=0H_0:\beta_k=0 vs H1:βk≠0H_1:\beta_k\neq0, with

t(β^k)=β^kse(β^k)t(\hat\beta_k)=\frac{\hat\beta_k}{se(\hat\beta_k)}

This is the t value column of R's summary(lm). Reject ⇒\Rightarrow the variable is significant; fail to reject ⇒\Rightarrow it has no significant effect on yy.

Careful (slides 42 and 43)

The t value column tests only H0:βk=0H_0:\beta_k=0. For any other null (βk=b≠0\beta_k=b\neq0) you must compute (β^k−b)/se(\hat\beta_k-b)/se yourself.

Activity 6 (parts 3): which variables are significant at 5%? (wage model, critical value 1.96)
Variable Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.385217 0.4218969 3.28 0.001
s 0.0981245 0.0108345 9.06 0.000
exp 0.0402309 0.0056855 7.08 0.000
IQ_test 0.0112204 0.0029018 3.87 0.000
age -0.0336168 0.0101734 -3.30 0.001
hours 0.0041505 0.0024687 1.68 0.093

R2=0.3011R^2=0.3011, adj. R2=0.2946R^2=0.2946, F=46.02F=46.02 on 5 and 534 df. Every variable has ∣t∣>1.96|t|>1.96 except hours (1.681.68): hours is not significant at 5%, but is at 10% (1.68>1.651.68>1.65).

What a p-value is

The p-value is the minimum significance level at which we would reject H0H_0 for the observed t-statistic:

p=P(∣T∣≥∣t(β^k)∣)\boxed{p=\mathbb{P}\big(|T|\ge |t(\hat\beta_k)|\big)}

i.e. the probability, if H0H_0 were true, of a t-statistic at least as extreme as the one we observed. Half of pp sits in each tail. Small pp = strong evidence against H0H_0.

Decision rule: reject H0H_0 at level α\alpha iff p<αp<\alpha.

p-value Reading Decision
0.5 Half of samples would look this extreme under H0H_0: weak evidence against H0H_0 Do not reject
0.072 Only 7.2% of samples this extreme: some evidence Not at 5%; would reject at 10%
0.014 Only 1.4%: evidence against H0H_0 Reject at 5%, not at 1%
  • Income-consumption slope: p<2×10−16p<2\times10^{-16}, so we reject at any significance level above that.
  • Housing bdrms: p=0.213p=0.213. Wage hours: p=0.093p=0.093 (the "." in the stars column marks significance at 10%).
Do not over-rely on p-values

The lecture stresses that p-values are widely misinterpreted and that leaning on them has contributed to reproducibility problems and p-hacking (trying specifications until p<0.05p<0.05). A p-value is not the probability that H0H_0 is true.


5. One-sided tests

Same t-statistic t(β^1)=(β^1−b)/se(β^1)t(\hat\beta_1)=(\hat\beta_1-b)/se(\hat\beta_1); only the rejection region changes: all of α\alpha goes in one tail, so the critical value is tα[df]t_{\alpha}[df] (the 1-tailed row of the table, e.g. t0.05[32]=1.694t_{0.05}[32]=1.694), not tα/2t_{\alpha/2}.

Left-tail test Right-tail test
Hypotheses H0:β≥bH_0:\beta\ge b vs H1:β<bH_1:\beta<b H0:β≤bH_0:\beta\le b vs H1:β>bH_1:\beta>b
Reject when t(β^)≤−tαt(\hat\beta)\le -t_\alpha t(β^)≥tαt(\hat\beta)\ge t_\alpha
Otherwise fail to reject fail to reject
Table trick (slide 58)

The t-table has a 1-tailed row and a 2-tailed row. For the same α\alpha use the 1-tailed row for one-sided tests (α=10%,5%,1%\alpha=10\%,5\%,1\% map to 1-tailed .10,.05,.01.10, .05, .01), and the 2-tailed row for two-sided tests.

Example: marginal propensity to consume (df 32, t0.05[32]=1.694t_{0.05}[32]=1.694)

H0H_0 H1H_1 tt Decision
β1≥0.7\beta_1\ge0.7 β1<0.7\beta_1<0.7 (0.81547−0.7)/0.01914=6.033(0.81547-0.7)/0.01914=6.033 Fail to reject (needs t≤−1.694t\le-1.694)
β1≤0.7\beta_1\le0.7 β1>0.7\beta_1>0.7 6.0336.033 Reject (since 6.033≥1.6946.033\ge1.694)
β1≥0.8\beta_1\ge0.8 β1<0.8\beta_1<0.8 0.8080.808 Fail to reject
β1≤0.8\beta_1\le0.8 β1>0.8\beta_1>0.8 0.8080.808 Fail to reject
Reconstructed from the slides: do the 0.8 results contradict each other?

The slide asks this but gives no answer on the page. They do not contradict: failing to reject β1≥0.8\beta_1\ge0.8 and failing to reject β1≤0.8\beta_1\le0.8 just says the data cannot separate β1\beta_1 from 0.8 in either direction (0.8080.808 is tiny relative to the critical value). Not rejecting is not accepting, so there is no inconsistency.


6. Summary of the testing procedure

  1. Write H0H_0 and H1H_1.
  2. Compute the t-statistic.
  3. Get the critical value for the chosen α\alpha (1-tailed or 2-tailed row, correct df).
  4. Compare and decide. Equivalent shortcut: reject iff p<αp<\alpha (two-sided: iff b∉b\notin the (1−α)(1-\alpha) CI).

7. Regression through the origin

Sometimes theory says y=0y=0 when x=0x=0, so the model has no intercept:

yi=β1xi+ui,y~i=β~1xiy_i=\beta_1x_i+u_i,\qquad \tilde y_i=\tilde\beta_1x_i

Both the theoretical and fitted lines pass through (0,0)(0,0).

OLS estimator

Minimise ∑(yi−β~1xi)2\sum(y_i-\tilde\beta_1x_i)^2. First-order condition:

∂∂β~1∑(yi−β~1xi)2=−2∑xi(yi−β~1xi)=0 ⇒ ∑u~ixi=0\frac{\partial}{\partial\tilde\beta_1}\sum(y_i-\tilde\beta_1x_i)^2=-2\sum x_i(y_i-\tilde\beta_1x_i)=0 \ \Rightarrow\ \sum \tilde u_ix_i=0
∑xiyi=β~1∑xi2⇒β~1=∑xiyi∑xi2\sum x_iy_i=\tilde\beta_1\sum x_i^2\quad\Rightarrow\quad\boxed{\tilde\beta_1=\frac{\sum x_iy_i}{\sum x_i^2}}
With intercept Through origin
Slope estimator β^1=∑(xi−xˉ)yi∑(xi−xˉ)2\hat\beta_1=\dfrac{\sum(x_i-\bar x)y_i}{\sum(x_i-\bar x)^2} β~1=∑xiyi∑xi2\tilde\beta_1=\dfrac{\sum x_iy_i}{\sum x_i^2}
Equal when only if xˉ=0\bar x=0
Unbiased when A1 to A4 hold (always, even if true β0=0\beta_0=0) only if the true intercept is β0=0\beta_0=0

Which mistake is worse?

Mistake Consequence
Include an intercept although true β0=0\beta_0=0 Minor: β^0\hat\beta_0 is just an estimate that should come out near 0; all estimates stay unbiased under A1 to A4.
Omit the intercept although true β0≠0\beta_0\neq0 Biased slope β~1\tilde\beta_1.

Direction of the bias (slides 72 to 73): if the true β0>0\beta_0>0, forcing the line through the origin makes it steeper, β~1>β^1\tilde\beta_1>\hat\beta_1; if β0<0\beta_0<0, it makes it flatter, β~1<β^1\tilde\beta_1<\hat\beta_1.

Lec06_through_origin Theory diagram (simulated points, not course data): the green no-intercept line pivots about the origin and distorts the slope in the direction opposite to the sign of the true intercept.

Activity 7 (part 4): HW3 Part A Q3, income-consumption

A colleague argues a negative intercept is impossible (consumption cannot be negative) and so fits a no-intercept model. Output shown on slide 75:

Lec06_data_hw3_regressions Real data: R output from slide 75, with-intercept model (top) and gdp + 0 model (bottom).

Model Slope s.e. Intercept Residual s.e. R2R^2
With intercept 0.8155 0.01914 -21890 (s.e. 1377, t = -15.9, p < 2e-16) 1325 on 32 df 0.9827
Through origin (gdp + 0) 0.51524 0.00928 none 3892 on 33 df 0.9894
Reconstructed from the slides: answers to (a) and (b)

(a) Yes: the intercept has t=−15.9t=-15.9, p<2×10−16p<2\times10^{-16}, so it is significantly different from zero. (b) Use the model with the intercept: the true intercept is clearly not 0, so omitting it biases the slope, and indeed the slope drops from 0.8155 to 0.5152, consistent with the β0<0⇒β~1<β^1\beta_0<0\Rightarrow\tilde\beta_1<\hat\beta_1 picture. The no-intercept model's higher printed R2R^2 (0.9894 vs 0.9827) is not comparable: R2R^2 is computed differently without an intercept, and its residual standard error (3892) is far worse than 1325.

The deck then shows "HW 4 Part A" (slide 76) with the sign pattern β0<0\beta_0<0 and β~1<β^1\tilde\beta_1<\hat\beta_1; the slide's regression output was not legible as text.

Should I include an intercept? (slide 77)

  • Theory (though it does not always give an answer).
  • Statistical tests: goodness of fit and a significance test on the intercept.

Summary

  • A hypothesis test asks how plausible the sample is if β=b\beta=b. Four steps: hypotheses, t=(β^−b)/set=(\hat\beta-b)/se, critical value, decision.
  • Two-sided: reject iff ∣t∣≥tα/2[df]|t|\ge t_{\alpha/2}[df]; equivalently iff bb lies outside the (1−α)(1-\alpha) CI.
  • The R t value column tests only βk=0\beta_k=0 (significance).
  • The p-value =P(∣T∣≥∣t∣)=\mathbb{P}(|T|\ge|t|) is the smallest α\alpha at which you reject; reject iff p<αp<\alpha. Handle with care (p-hacking).
  • One-sided tests use the same tt but a one-tail critical value tαt_\alpha; "fail to reject" both sides is not a contradiction.
  • Dropping the intercept when the true one is non-zero biases the slope; including a needless one is harmless.