PS2

  1. 1

    Load and inspect the insurance dataset. Load the insurance dataset using kagglehub and explore its basic properties. This is the same EDA workflow you followed in HW1: load, inspect, understand. (Dataset identifier mirichoi0218/insurance, file insurance.csv. Display the shape, then df_ins.info() and df_ins.describe().)

    Setup cell used for the whole notebook:

    python
    pip install -q kagglehub
    
    import pandas as pd
    import numpy as np
    import matplotlib.pyplot as plt
    import seaborn as sns
    import kagglehub
    import os
    from sklearn.model_selection import train_test_split, cross_val_score
    from sklearn.linear_model import LinearRegression, Ridge, Lasso
    from sklearn.preprocessing import PolynomialFeatures
    from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
    
  2. 2

    Missing values and the target distribution. Check for missing values and visualise the distribution of charges (our target variable). Even when you expect clean data, always verify.

    Conceptual: The insurance dataset has zero missing values. Does that mean we can skip the data quality step in real projects? Why is checking for missing values always the first thing you do? (2 to 3 sentences)

  3. 3

    Feature-target relationships. Create visualisations to understand how each feature relates to charges. This helps you build intuition before modelling: which features will be useful predictors?

    Conceptual: Based on these four plots, which feature appears to have the strongest relationship with charges? Do you notice anything unusual in the age-vs-charges scatter plot (look for separate "lanes" or groups of points)? What might explain this pattern? (3 to 4 sentences)

  4. 4

    Correlation analysis. Compute the correlation matrix for numeric features. Remember: correlation only captures linear relationships between numeric variables; it can miss important patterns.

    Conceptual: Which numeric feature has the highest correlation with charges? Is correlation capturing the full picture? (Hint: think about smoker, which is categorical and does not appear in the correlation matrix.) (2 to 3 sentences)

  5. 5

    Encode categorical variables. Before we can feed the data into a regression model, we need to convert categorical columns to numbers. You did this in HW1 with pd.get_dummies(): same idea here.

  6. 6

    Define features and split. Separate the features (X) from the target (y) and split into training and test sets. Same logic as HW1: we never test on training data. (Use test_size=0.2, random_state=42.)

    Conceptual: Why is it critical to evaluate your model on data it has never seen during training? What would happen if you tested on your training data? (2 to 3 sentences)

  7. 7

    Simple linear regression (one feature). Start with the simplest possible model: predict charges using only age. This gives you a baseline to beat.

  8. 8

    Multiple linear regression (all features). Now use all available features. More information should mean better predictions: let's see by how much.

    Conceptual: R2R^2 jumped significantly from simple to multiple regression. What does this tell you about the importance of using multiple features? Is an R2R^2 of about 0.75 "good enough" for an insurance pricing model? What are the business implications of prediction errors? (3 to 4 sentences)

  9. 9

    Polynomial regression, power and danger. Linear regression assumes straight-line relationships. Polynomial features let the model capture curves and interactions, but too much flexibility leads to overfitting.

    Heads up (from the sheet): In step 2 (degree 5), the model generates over 1,200 features from only about 1,070 training samples. Don't be alarmed if the test R2R^2 is wildly negative: that means the predictions are worse than guessing the average.

    Conceptual: Compare the degree-5 model's R2R^2 on training data versus test data. What do you observe? What is this phenomenon called, and why is it dangerous in a real business setting? (2 to 3 sentences)

  10. 10

    Ridge and Lasso regularisation. Regularisation adds a penalty for large coefficients, preventing the model from overfitting. Ridge shrinks all coefficients; Lasso can set some to exactly zero (automatic feature selection). Use the degree-2 polynomial features from Q9.

    Conceptual: Compare the two Lasso models. What happened when you increased alpha from 0.01 to 1.0: how did R2R^2 and the number of zero coefficients change? What does Lasso do that Ridge doesn't? Why might automatic feature selection be valuable when you have many features? (3 to 4 sentences)

  11. 11

    Cross-validation. A single train/test split can be lucky or unlucky. Cross-validation gives you a more reliable estimate by testing on multiple different splits. (Use the multiple linear regression model, not the polynomial one; pass the full dataset, cv=5, scoring='r2'.)

    Conceptual: Why is the cross-validation R2R^2 more trustworthy than a single train/test split R2R^2? What does a low standard deviation across folds tell you about your model? (2 to 3 sentences)

  12. 12

    Model comparison. Put all your models side by side to see which one performs best on the test set. (Poly-2, Ridge and Lasso all use the same degree-2 polynomial features: Ridge and Lasso are regularised versions of the polynomial model, not entirely independent models. Simple LR uses only age, and Multiple LR uses the original encoded features. Build a DataFrame with columns Model and Test R² for all five, using the better Lasso, alpha 0.01, then a bar chart.)

Related: Lecture 4 - Regression, Lecture 2 - EDA and Charts, Lecture 3 - Classification, KNN, Random Tree, Lecture 5 - Clustering and Dimensionality Reduction