Goodness of Fit & Multicollinearity
- #intro-econometrics
- #goodness-of-fit
- #r-squared
- #adjusted-r-squared
- #multicollinearity
- #OLS
- #variance-of-estimator
Goodness of Fit & Multicollinearity
Part of: Intro Econometrics | Key concepts: R-squared, Adjusted R-squared, Total Sum of Squares, Multicollinearity, Variance of the OLS Estimator, F-test
Where this lecture sits
The thread from last weekLast week was about joint hypothesis testing with the F-test: build a restricted and an unrestricted model, compare their fit, and ask whether the extra variables in the unrestricted model add anything. This week pushes on the same idea — how well does a model fit? — but turns it into a single summary number, R-squared, and then studies a problem that quietly inflates the variance of our estimates: multicollinearity.
The lecture has two halves:
- Goodness-of-fit — defining , deriving it from the way OLS splits the variance of , its limitations, and the fix (adjusted ). Plus a neat re-write of the F-statistic purely in terms of .
- Multicollinearity — what happens when the explanatory variables are linearly related to each other. Perfect collinearity makes estimation impossible; partial collinearity makes it imprecise.
Part 1 — Goodness-of-Fit: the statistic
The basic idea
Every fitted OLS model splits each observed outcome into two pieces:
where is the predicted value. A good model makes the explained part big and the residual part small. turns "big" and "small" into a percentage.
The intuition in one line= the fraction of the variance in that the model accounts for. The closer to 1, the more of the ups-and-downs in your 's capture.
Two algebraic properties of OLS we need
To define cleanly we first need two facts that fall straight out of the OLS first-order conditions (FOCs). Recall that minimising gives FOCs:
The very first one — — comes from the intercept , and it is the workhorse of everything below.
Property (1): the predicted values have the same mean asDerivation: sum over all and divide by : The last term is zero by the intercept FOC (), so . The regression line passes through the point of sample means.
Property (2): predicted values and residuals are uncorrelatedDerivation sketch: write . Substituting and pulling the constants out of the sum, every term becomes one of the FOC sums (, ), all of which equal zero. Hence the covariance is exactly zero — by construction, not by luck.
Partitioning the variance
Because the predicted values and residuals have zero covariance, the variance of splits cleanly:
Multiplying the sample-variance version through by gives the sum-of-squares decomposition:
| Term | Name | Meaning |
|---|---|---|
| TSS — Total Sum of Squares | Total variation in the outcome | |
| MSS — Model Sum of Squares | Variation the model explains | |
| RSS — Residual Sum of Squares | Variation the model leaves unexplained |
Where Property (1) sneaks inThe MSS term is centred on , not . We are only allowed to do that because Property (1) told us . Without it, the cross-term wouldn't vanish and the tidy decomposition would fall apart.
Defining
Divide the decomposition through by (equivalently by TSS):
The first fraction is the explained share — that's :
Because the two shares add to 1 and both are non-negative, and we can read it as a percentage: how much of the variance in the dependent variable the model explains.
Reading in practiceIncome–consumption model: → income explains 98.3% of the variance in consumption; only 1.7% is unexplained. Housing prices (
price ~ bdrms + sqrft): → area and bedrooms explain 63.2% of the variance in apartment prices; 36.8% is left to everything else.
Call: lm(formula = price ~ bdrms + sqrft, data = mydata)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -19315.00 31046.62 -0.622 0.536
bdrms 15198.19 9483.52 1.603 0.113
sqrft 128.44 13.82 9.291 1.4e-14 ***
Multiple R-squared: 0.6319, Adjusted R-squared: 0.6233
F-statistic: 72.96 on 2 and 85 DF, p-value: < 2.2e-16
The boundary cases
When is or ?
- ⟺ all residuals are zero, i.e. every point lies exactly on the regression surface. Random error always exists, so this never truly happens — not even if you observed the entire population.
- ⟺ the model has no explanatory power at all: the 's tell you nothing about . A model without meaning.
Why can mislead
depends on the variance of the outcomeis sensitive to how spread-out is. If is large, can look small even when the model fits well; if is a smooth fiscal aggregate it can look huge.
- Wage models rarely break ~50% — individual wages are noisy.
- Fiscal series (GDP, consumption, investment, demand) routinely exceed 90% — they move slowly and predictably.
⟹ You may only compare across models if the outcome variable is identical. A model of and a model of are not comparable on .
Adjusted — comparing models of different size
The mechanical flaw inAdding any regressor — even pure noise — weakly increases . Why? The new variable can only shrink the RSS (OLS will set its coefficient to 0 in the worst case, never making the fit worse). So rewards throwing variables at the model, which is not what we want.
The fix penalises the model for each extra parameter, using degrees of freedom:
How to read itAdjusted rises only if a new variable improves fit by more than would be expected from a random regressor. If you add junk, falls even though rises. It is the right tool for choosing between models with different numbers of explanatory variables.
Simulation: adding a noise variableTrue model . Now add an unrelated noise variable :
- increases (mechanically, as always)
- Adjusted decreases — correctly flagging that added nothing.
Limits that apply to both measuresNeither nor can compare models with different outcome variables or models with no intercept. Adjusted buys you only one extra thing: comparison across different numbers of regressors.
| Adjusted | ||
|---|---|---|
| Range | Can be negative | |
| Adding a regressor | Never decreases | Decreases if the variable is useless |
| Compare different # of regressors? | ❌ No | ✅ Yes |
| Compare different outcome variables? | ❌ No | ❌ No |
| Compare models without intercept? | ❌ No | ❌ No |
| Best use | Single-model fit summary | Model selection |
The F-statistic, rewritten in terms of
The F-test for restrictions started life as a ratio of residual sums of squares:
Divide numerator and denominator by and use together with :
The hidden assumptionThis clean form assumes is identical in the restricted and unrestricted models. That holds when the restriction is (same dependent variable), but fails for restrictions of the form with — there you must go back to the RSS version.
Testing whether the whole model is significant
A special case: does the model as a whole explain anything? Restrict every slope to zero.
- Unrestricted:
- Restricted: (so )
- Null:
Plugging into the formula:
This is exactly the F-statistic printed at the bottom of every lm summary. A small p-value means at least one regressor matters.
Wage model — overall significance
ln_wage ~ s + exp + IQ_test + age + hours: , F = 46.02 on 5 and 534 df, p < 2.2e-16. The model is jointly highly significant — reject that none of the variables explain wages.
Part 2 — Multicollinearity
Multicollinearity is a linear relationship among the explanatory variables themselves. It comes in two flavours, with very different consequences.
Perfect multicollinearity — estimation is impossible
DefinitionOne regressor is an exact linear function of one or more others, e.g. , or , or . Then OLS cannot be estimated.
Crucially, this is about linear dependence only. Variables can be strongly — even perfectly — correlated in a non-linear way and still be fine:
- → fine
- → fine (a quadratic is not a linear transformation)
Where it really comes from: misspecificationPerfect collinearity is almost always a modelling mistake, such as:
- the same variable entered twice in different units (age in months and age in years),
- two variables linked by an exact identity (age in years and year of birth).
R's
lmsilently drops one of the offending variables and reportsNA:Coefficients: (1 not defined because of singularities) Estimate Std. Error t value Pr(>|t|) educ 0.058048 0.006694 8.672 < 2e-16 *** age 0.024560 0.004878 5.035 6.2e-07 *** y_birth NA NA NA NA
Why it breaks — the FOC argumentSuppose . The OLS FOCs include Substituting into the first gives — which is just the negative of the second FOC, carrying no new information. We are left with independent equations for unknowns: an underdetermined system with infinitely many solutions. No unique exists.
Partial multicollinearity — estimation is possible but imprecise
When the 's are correlated but not perfectly, none of the Gauss–Markov assumptions break. OLS stays BLUE (unbiased, minimum variance among linear unbiased estimators). The catch:
The cost of partial collinearityThe estimator's variance — though still minimal in the BLUE sense — is larger the more the regressors are correlated. Bigger variance ⟹ bigger standard errors ⟹ smaller t-stats ⟹ harder to find anything "significant", even when the true effect is real.
The variance-of-the-estimator formula
In a single-variable regression:
In a multivariate regression the formula gains one extra factor:
where is the from regressing on all the other explanatory variables:
How to read each piece of the formula
- ↑ → more noise in → larger variance.
- ↑ → more spread in the regressor → smaller variance (more leverage to identify the slope).
- ↑ → is well explained by the other regressors (high collinearity) → shrinks → variance explodes. This is the multicollinearity penalty. is the Variance Inflation Factor.
Activity 8 — the two extremes ofModel .
- : is a perfect linear function of — perfect multicollinearity. , so is undefined (division by zero). This matches the "impossible to estimate" result above.
- : zero collinearity between and . The variance of is the same as in the simple regression (only the error's standard deviation may differ).
The classic bias–variance / fit–precision trade-offAdding regressors tends to raise for the variables already in the model (they get more correlated company), so each estimate's variance climbs and its significance falls — even as the overall model fit () improves. More variables = better fit but noisier individual coefficients.
Rule of thumb and diagnosis
Spotting partial multicollinearitylives in . Exactly 0 or exactly 1 are both rare. The practical danger zone is close to 1 — rule of thumb > 0.75–0.8. Diagnosis is cheap: before running the regression, inspect the correlation matrix of the explanatory variables.
Worked example: parental education in a wage regression
The modelwhere
meduc/feducare mother's/father's years of schooling. Question: does family background affect wages?
Step 1 — the full regression. Both parental-education coefficients look insignificant:
Estimate Std. Error t value Pr(>|t|)
educ 0.042323 0.009237 4.582 5.5e-06 ***
age 0.016799 0.005709 2.943 0.00337 **
exper 0.015232 0.004584 3.323 0.00094 ***
IQ 0.005082 0.001188 4.279 2.2e-05 ***
meduc 0.017660 0.009823 1.798 0.07267 .
feduc 0.001900 0.009325 0.204 0.83859
Multiple R-squared: 0.1965, Adjusted R-squared: 0.1891
It is tempting to drop both and conclude "family background doesn't matter." Don't — test them jointly first.
Step 2 — the joint F-test. , via anova(full, restricted):
Model 1: lwage ~ educ + age + exper + IQ + meduc + feduc
Model 2: lwage ~ educ + age + exper + IQ
Res.Df RSS Df Sum of Sq F Pr(>F)
1 655 90.381
2 657 92.283 -2 -1.9017 6.8908 0.00109 **
The joint test rejects at 1%. So we cannot remove both — together they matter, even though neither is individually significant.
The t-test vs F-test contradiction — and its causeEach individual t-test says "insignificant"; the joint F-test says "significant." This is the textbook fingerprint of multicollinearity. The correlation matrix reveals why:
feduc meduc feduc 1.00 0.88 meduc 0.88 1.00Mother's and father's education are correlated at 0.88. Each inflates the other's standard error (via the factor), drowning their individual t-stats — yet jointly they clearly explain wages.
Step 3 — confirm by dropping one. Remove feduc and meduc jumps to significant; remove meduc and feduc jumps to significant. Each only looked weak because of the other's presence.
The practical conclusionDon't include two near-identical regressors. Use only
meduc, or onlyfeduc, or their average as a single "parental education" measure. The clean result: each year of a parent's education raises the child's wage by ~2%.
Practice exam: the income–consumption model
Setup34 yearly observations. , . Estimates: , , , .
Q1 — Predicted consumption at average income? By Property (1), the OLS line passes through the sample means. So at the prediction is exactly — no arithmetic needed.
Q2 — Does the line pass through the origin? vs .
⟹ reject : the intercept is significantly non-zero, so the line does not pass through the origin.
Q3 — Is the marginal propensity to consume below 90%? One-sided: vs .
⟹ reject : the MPC is significantly less than 0.9.
Q4 — Re-estimate in NIS instead of USD (exchange rate 3.5 NIS/USD), so both variables are scaled by 3.5: .
Scaling both sides leaves the slope (and its se) unchangedMultiplying and by the same constant cancels in the slope: . So and — both identical. (Only the intercept rescales: .)
Summary
- OLS splits every into a predicted part and a residual . Two FOC-driven properties — and — let us split the variance: TSS = MSS + RSS.
- is the share of variance explained, . It depends on , so only compare it across models with the same outcome.
- always rises when you add a variable, even noise. Adjusted penalises extra parameters and is the right tool for comparing models of different size (but still not different outcomes / no-intercept models).
- The F-statistic can be written purely in : , valid when is unchanged. Setting gives the overall model significance test reported by
lm. - Perfect multicollinearity (exact linear dependence) makes the FOCs redundant → no unique solution → estimation impossible. It's almost always a misspecification.
- Partial multicollinearity keeps OLS unbiased but inflates variance via . High (rule of thumb > 0.75–0.8) → tiny t-stats. The tell-tale sign is insignificant t-tests but a significant joint F-test (the parental-education example). Diagnose with the correlation matrix; fix by combining or dropping redundant regressors.
Related Notes
- Previous: Lec_07-Multiple Hypothesis Tests — joint hypotheses and the F-test that this lecture re-expresses through
- Next: Lec_09-Model Misspecification & Quadratics, then Lec_10-Dummy Variables and Lec_12-Heteroskedasticity
- Foundational: F-test, OLS Estimation, Variance of the OLS Estimator, Gauss-Markov Theorem
- Concepts: R-squared, Adjusted R-squared, Multicollinearity, Total Sum of Squares
- Hub: Intro Econometrics