PS2
- #data-science
- #intro-data-science
- #regression
- #linear-regression
- #polynomial-regression
- #ridge-lasso
- #cross-validation
- #eda
Toolkit
- 1
Load and inspect the insurance dataset. Load the insurance dataset using
kagglehuband explore its basic properties. This is the same EDA workflow you followed in HW1: load, inspect, understand. (Dataset identifiermirichoi0218/insurance, fileinsurance.csv. Display the shape, thendf_ins.info()anddf_ins.describe().)Setup cell used for the whole notebook:
pip install -q kagglehub import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns import kagglehub import os from sklearn.model_selection import train_test_split, cross_val_score from sklearn.linear_model import LinearRegression, Ridge, Lasso from sklearn.preprocessing import PolynomialFeatures from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score - 2
Missing values and the target distribution. Check for missing values and visualise the distribution of
charges(our target variable). Even when you expect clean data, always verify.Conceptual: The insurance dataset has zero missing values. Does that mean we can skip the data quality step in real projects? Why is checking for missing values always the first thing you do? (2 to 3 sentences)
- 3
Feature-target relationships. Create visualisations to understand how each feature relates to charges. This helps you build intuition before modelling: which features will be useful predictors?
Conceptual: Based on these four plots, which feature appears to have the strongest relationship with charges? Do you notice anything unusual in the age-vs-charges scatter plot (look for separate "lanes" or groups of points)? What might explain this pattern? (3 to 4 sentences)
- 4
Correlation analysis. Compute the correlation matrix for numeric features. Remember: correlation only captures linear relationships between numeric variables; it can miss important patterns.
Conceptual: Which numeric feature has the highest correlation with charges? Is correlation capturing the full picture? (Hint: think about
smoker, which is categorical and does not appear in the correlation matrix.) (2 to 3 sentences) - 5
Encode categorical variables. Before we can feed the data into a regression model, we need to convert categorical columns to numbers. You did this in HW1 with
pd.get_dummies(): same idea here. - 6
Define features and split. Separate the features (X) from the target (y) and split into training and test sets. Same logic as HW1: we never test on training data. (Use
test_size=0.2,random_state=42.)Conceptual: Why is it critical to evaluate your model on data it has never seen during training? What would happen if you tested on your training data? (2 to 3 sentences)
- 7
Simple linear regression (one feature). Start with the simplest possible model: predict charges using only
age. This gives you a baseline to beat. - 8
Multiple linear regression (all features). Now use all available features. More information should mean better predictions: let's see by how much.
Conceptual: jumped significantly from simple to multiple regression. What does this tell you about the importance of using multiple features? Is an of about 0.75 "good enough" for an insurance pricing model? What are the business implications of prediction errors? (3 to 4 sentences)
- 9
Polynomial regression, power and danger. Linear regression assumes straight-line relationships. Polynomial features let the model capture curves and interactions, but too much flexibility leads to overfitting.
Heads up (from the sheet): In step 2 (degree 5), the model generates over 1,200 features from only about 1,070 training samples. Don't be alarmed if the test is wildly negative: that means the predictions are worse than guessing the average.
Conceptual: Compare the degree-5 model's on training data versus test data. What do you observe? What is this phenomenon called, and why is it dangerous in a real business setting? (2 to 3 sentences)
- 10
Ridge and Lasso regularisation. Regularisation adds a penalty for large coefficients, preventing the model from overfitting. Ridge shrinks all coefficients; Lasso can set some to exactly zero (automatic feature selection). Use the degree-2 polynomial features from Q9.
Conceptual: Compare the two Lasso models. What happened when you increased alpha from 0.01 to 1.0: how did and the number of zero coefficients change? What does Lasso do that Ridge doesn't? Why might automatic feature selection be valuable when you have many features? (3 to 4 sentences)
- 11
Cross-validation. A single train/test split can be lucky or unlucky. Cross-validation gives you a more reliable estimate by testing on multiple different splits. (Use the multiple linear regression model, not the polynomial one; pass the full dataset,
cv=5,scoring='r2'.)Conceptual: Why is the cross-validation more trustworthy than a single train/test split ? What does a low standard deviation across folds tell you about your model? (2 to 3 sentences)
- 12
Model comparison. Put all your models side by side to see which one performs best on the test set. (Poly-2, Ridge and Lasso all use the same degree-2 polynomial features: Ridge and Lasso are regularised versions of the polynomial model, not entirely independent models. Simple LR uses only age, and Multiple LR uses the original encoded features. Build a DataFrame with columns
ModelandTest R²for all five, using the better Lasso, alpha 0.01, then a bar chart.)
Related: Lecture 4 - Regression, Lecture 2 - EDA and Charts, Lecture 3 - Classification, KNN, Random Tree, Lecture 5 - Clustering and Dimensionality Reduction