{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recitation 4: Regression with scikit-learn\n",
    "### Data Science for Entrepreneurship-Economy | Spring 2026\n",
    "\n",
    "**Today's goal**: Build and compare regression models that predict a continuous number — from a single feature to polynomial curves to regularized models.\n",
    "\n",
    "By the end of this session you'll be able to:\n",
    "- Build simple and multiple linear regression models\n",
    "- Interpret coefficients as business-meaningful \"price tags\"\n",
    "- Fit polynomial curves and diagnose overfitting\n",
    "- Apply Ridge and Lasso regularization\n",
    "- Use cross-validation for reliable model evaluation\n",
    "\n",
    "**Time**: ~90 minutes | **Exercises**: 2 | **Format**: Guided walkthrough — follow along, then practice\n",
    "\n",
    "**How this session works**: Same as Recitation 3 — the instructor drives the pipeline, you run each cell, and at key decision points you'll get exercises."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📖 Quick Recap\n",
    "\n",
    "- In Recitation 3 you built classifiers that predict **labels** (churn / no-show). Today you predict **numbers**.\n",
    "- The scikit-learn pattern is identical: `create → .fit() → .predict()`\n",
    "- New metrics: MAE, RMSE, R² replace accuracy/precision/recall\n",
    "- New tools: polynomial features, Ridge, Lasso, cross-validation\n",
    "- The thinking is the same: prepare data → train → evaluate → compare → decide\n",
    "\n",
    "Last week you predicted *yes/no*. This week you predict *how much*."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# scikit-learn imports for regression\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
    "from sklearn.preprocessing import PolynomialFeatures, StandardScaler\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score\n",
    "\n",
    "sns.set_style(\"whitegrid\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Load the AutoBid used-car dataset\n",
    "df = pd.read_csv(\"autobid_used_cars.csv\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **AutoBid** is an Israeli online used-car marketplace. The platform connects private sellers with buyers, but sellers often struggle to price their cars correctly — overpricing leads to weeks without offers, underpricing leaves money on the table. AutoBid's product lead, Dana, wants to build a **price prediction engine** that suggests a fair listing price based on car attributes. This would help sellers list faster and help buyers trust that prices are fair.\n",
    "\n",
    "Dana needs a pricing engine that sellers trust. Let's build one."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "print(f\"Dataset: {df.shape[0]} listings × {df.shape[1]} columns\\n\")\n",
    "print(df.describe().round(1))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "df[\"listing_price_thousands\"].hist(bins=20, color=\"#214073\", edgecolor=\"white\")\n",
    "plt.xlabel(\"Listing Price (₪K)\")\n",
    "plt.ylabel(\"Number of Cars\")\n",
    "plt.title(\"Distribution of Used Car Prices — AutoBid\")\n",
    "plt.show()\n",
    "\n",
    "print(f\"Mean price: ₪{df['listing_price_thousands'].mean():.0f}K\")\n",
    "print(f\"Median price: ₪{df['listing_price_thousands'].median():.0f}K\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right-skewed — most cars cluster in the middle range, with a tail of luxury and electric vehicles stretching higher. Mean > median confirms the skew."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n",
    "\n",
    "axes[0].scatter(df[\"car_age_years\"], df[\"listing_price_thousands\"], alpha=0.4, color=\"#214073\")\n",
    "axes[0].set_xlabel(\"Car Age (years)\")\n",
    "axes[0].set_ylabel(\"Price (₪K)\")\n",
    "axes[0].set_title(\"Age vs. Price — Linear?\")\n",
    "\n",
    "axes[1].scatter(df[\"km_driven_thousands\"], df[\"listing_price_thousands\"], alpha=0.4, color=\"#008C8C\")\n",
    "axes[1].set_xlabel(\"KM Driven (thousands)\")\n",
    "axes[1].set_ylabel(\"Price (₪K)\")\n",
    "axes[1].set_title(\"KM vs. Price\")\n",
    "\n",
    "axes[2].scatter(df[\"engine_size_cc\"], df[\"listing_price_thousands\"], alpha=0.4, color=\"#E6592D\")\n",
    "axes[2].set_xlabel(\"Engine Size (cc)\")\n",
    "axes[2].set_ylabel(\"Price (₪K)\")\n",
    "axes[2].set_title(\"Engine Size vs. Price — Curved?\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Age vs. price is clearly linear (downward). Engine size vs. price shows a curve — rises steeply then flattens. That nonlinearity will matter later."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "print(\"Correlation between car_age and km_driven:\")\n",
    "print(f\"  r = {df['car_age_years'].corr(df['km_driven_thousands']):.2f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Older cars have been driven more — these two features are tangled (multicollinearity). We'll see how this affects our model in later sections."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "X = df.drop(columns=[\"listing_price_thousands\", \"listing_id\"])\n",
    "y = df[\"listing_price_thousands\"]\n",
    "\n",
    "# One-hot encode categorical columns (BMW is baseline — dropped first alphabetically)\n",
    "X = pd.get_dummies(X, columns=[\"fuel_type\", \"car_brand\", \"city\"], drop_first=True)\n",
    "print(f\"Features after encoding: {X.shape[1]} columns\")\n",
    "\n",
    "# Train/test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=42\n",
    ")\n",
    "print(f\"Training: {X_train.shape[0]} | Test: {X_test.shape[0]}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 📏 Section 2: Simple Linear Regression\n",
    "*One feature, one line. How far can car age alone take us?*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Use ONLY car_age_years\n",
    "X_train_age = X_train[[\"car_age_years\"]]\n",
    "X_test_age = X_test[[\"car_age_years\"]]\n",
    "\n",
    "simple_model = LinearRegression()\n",
    "simple_model.fit(X_train_age, y_train)\n",
    "\n",
    "print(f\"Slope (β₁):     {simple_model.coef_[0]:.2f}\")\n",
    "print(f\"Intercept (β₀): {simple_model.intercept_:.2f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📖 Interpreting the Coefficients\n",
    "\n",
    "Read the slope as depreciation per year: each additional year of age drops the predicted price. The intercept is the predicted price of a brand-new car (age = 0).\n",
    "\n",
    "Notice the slope absorbs the effect of km_driven too — older cars have more km, and since km isn't in the model, age takes credit for both effects. We'll see this change in multiple regression."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "y_pred_simple = simple_model.predict(X_test_age)\n",
    "\n",
    "r2_simple = r2_score(y_test, y_pred_simple)\n",
    "rmse_simple = np.sqrt(mean_squared_error(y_test, y_pred_simple))\n",
    "mae_simple = mean_absolute_error(y_test, y_pred_simple)\n",
    "\n",
    "print(f\"Simple Regression (car_age only):\")\n",
    "print(f\"  R²:   {r2_simple:.3f}\")\n",
    "print(f\"  RMSE: ₪{rmse_simple:.1f}K\")\n",
    "print(f\"  MAE:  ₪{mae_simple:.1f}K\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "plt.scatter(X_test_age, y_test, alpha=0.5, label=\"Actual\", color=\"#214073\")\n",
    "age_sorted = np.sort(X_test_age.values, axis=0)  # sorted for a clean line\n",
    "plt.plot(age_sorted, simple_model.predict(age_sorted),\n",
    "         color=\"#E6592D\", linewidth=2, label=\"Prediction line\")\n",
    "plt.xlabel(\"Car Age (years)\")\n",
    "plt.ylabel(\"Price (₪K)\")\n",
    "plt.title(f\"Simple Regression | R² = {r2_simple:.2f}\")\n",
    "plt.legend()\n",
    "plt.show()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🎯 Exercise 1: Predict and Interpret\n",
    "\n",
    "- A seller has a **7-year-old car**. Using the simple model's equation (slope and intercept from above), calculate the predicted price **by hand** (mental math).\n",
    "- Then verify with `simple_model.predict([[7]])`.\n",
    "- The actual listing price is ₪82K. What's the residual? Did the model over- or underestimate?\n",
    "- Is R² good enough for Dana's pricing engine? Why or why not?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "##### Your code goes here #####\n",
    "\n",
    "# 1. Manual calculation: Price = intercept + slope × 7\n",
    "\n",
    "# 2. Verify with model\n",
    "\n",
    "# 3. Residual = actual - predicted. Over or under?\n",
    "\n",
    "# 4. Is this R² good enough? (comment)\n",
    "\n",
    "###### End of Your Code ######"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ✅ Solution\n",
    "\n",
    "<details>\n",
    "<summary>Click to reveal</summary>\n",
    "\n",
    "```python\n",
    "# Manual: intercept + slope * 7\n",
    "manual_pred = simple_model.intercept_ + simple_model.coef_[0] * 7\n",
    "print(f\"Manual prediction: ₪{manual_pred:.1f}K\")\n",
    "\n",
    "# Verify\n",
    "model_pred = simple_model.predict([[7]])[0]\n",
    "print(f\"Model prediction:  ₪{model_pred:.1f}K\")\n",
    "\n",
    "# Residual\n",
    "actual = 82\n",
    "residual = actual - model_pred\n",
    "print(f\"Residual: {residual:+.1f}K ({'underestimated' if residual > 0 else 'overestimated'})\")\n",
    "\n",
    "# R² ~ 0.55 means 45% of price variation is unexplained.\n",
    "# RMSE ~ ₪34K means typical errors are huge — not reliable enough for a pricing tool.\n",
    "print(f\"\\nR² = {r2_simple:.2f} — too rough for a pricing engine. Need more features.\")\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 💡 Insight\n",
    "\n",
    "Car age alone explains about half the price variation. A typical error of ₪30K+ is too big for a pricing tool — a seller following this would misprice significantly. We need more features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 📊 Section 3: Multiple Regression\n",
    "*Add all features. Every variable gets its own \"price tag.\"*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "full_model = LinearRegression()\n",
    "full_model.fit(X_train, y_train)\n",
    "y_pred_full = full_model.predict(X_test)\n",
    "\n",
    "r2_full = r2_score(y_test, y_pred_full)\n",
    "rmse_full = np.sqrt(mean_squared_error(y_test, y_pred_full))\n",
    "\n",
    "print(f\"Multiple Regression (all features):\")\n",
    "print(f\"  R²:   {r2_full:.3f}\")\n",
    "print(f\"  RMSE: ₪{rmse_full:.1f}K\")\n",
    "print(f\"\\nImprovement from simple: R² {r2_simple:.2f} → {r2_full:.2f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "coefs = pd.Series(full_model.coef_, index=X_train.columns)\n",
    "coefs[\"Intercept\"] = full_model.intercept_\n",
    "\n",
    "# Show top coefficients by absolute value\n",
    "print(\"Top coefficients (sorted by |value|):\\n\")\n",
    "top = coefs.drop(\"Intercept\").abs().sort_values(ascending=False).head(10)\n",
    "for feat in top.index:\n",
    "    print(f\"  {feat:30s}: {coefs[feat]:+.2f} ₪K\")\n",
    "print(f\"\\n  {'Intercept':30s}: {coefs['Intercept']:.2f} ₪K\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📖 Interpreting the Coefficients\n",
    "\n",
    "Read each coefficient as: \"all else equal, this feature changes the predicted price by this much.\"\n",
    "\n",
    "Since BMW is the **baseline brand** (dropped by one-hot encoding), the brand coefficients show how much cheaper each brand is compared to BMW. For example, `car_brand_Hyundai: −₪44K` means a Hyundai is predicted to be ₪44K cheaper than an equivalent BMW. This confirms BMW's premium status.\n",
    "\n",
    "Similarly, `fuel_type_Electric: +₪35K` means electric cars are ₪35K more expensive than the baseline fuel type (Diesel).\n",
    "\n",
    "Notice that `car_age_years` coefficient shrank compared to simple regression. That's because `km_driven_thousands` now absorbs some of what age was explaining alone — they share information."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Compare age coefficient: simple vs. multiple\n",
    "print(f\"Simple model — car_age slope: {simple_model.coef_[0]:+.2f}\")\n",
    "print(f\"Multiple model — car_age:     {coefs['car_age_years']:+.2f}\")\n",
    "print(f\"Multiple model — km_driven:   {coefs['km_driven_thousands']:+.4f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 💬 Instructor Discussion: The Multicollinearity Question\n",
    "\n",
    "> **Discuss with the class:**\n",
    ">\n",
    "> - Why did the `car_age_years` coefficient shrink when we added all features?\n",
    "> - If you removed `km_driven_thousands`, what would happen to the age coefficient?\n",
    "> - Should Dana include BOTH age and km, or just one?\n",
    "\n",
    "<details>\n",
    "<summary>Key points</summary>\n",
    "\n",
    "- `car_age` shrank because `km_driven` now captures some of the same effect. These features are correlated (r ≈ 0.75) — they share information, and the shared variance gets split between them.\n",
    "- If we removed `km_driven`, `car_age` would grow back toward its simple regression value, absorbing the km effect again.\n",
    "- Keep both — the model's R² is higher with both features. Regularization (Section 5) handles the multicollinearity more elegantly.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "plt.figure(figsize=(7, 7))\n",
    "plt.scatter(y_test, y_pred_full, alpha=0.5, color=\"#214073\")\n",
    "plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()],\n",
    "         \"r--\", linewidth=2, label=\"Perfect prediction\")\n",
    "plt.xlabel(\"Actual Price (₪K)\")\n",
    "plt.ylabel(\"Predicted Price (₪K)\")\n",
    "plt.title(f\"Multiple Regression — Actual vs. Predicted | R² = {r2_full:.2f}\")\n",
    "plt.legend()\n",
    "plt.show()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 💡 Insight\n",
    "\n",
    "R² jumped substantially — adding brand, fuel type, km, and other features captured much of the remaining variation. Points cluster closer to the red line. For a rough pricing tool, this is getting usable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 🔄 Section 4: Polynomial Regression\n",
    "*When straight lines aren't enough — fitting curves to capture nonlinear patterns.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📖 Why Polynomial?\n",
    "\n",
    "Remember the engine_size scatter plot from Section 1? Price rises steeply for small engines, then flattens for large ones. A straight line can't capture that curve.\n",
    "\n",
    "Polynomial regression adds x² (and higher powers) as features so the model can fit curves. We'll apply it **to engine_size specifically** — because that's the feature with the visible nonlinearity. Adding polynomial terms to every feature would create dozens of extra columns that just add noise."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# PolynomialFeatures on engine_size only\n",
    "poly = PolynomialFeatures(degree=2, include_bias=False)\n",
    "engine_train = poly.fit_transform(X_train[[\"engine_size_cc\"]])\n",
    "engine_test = poly.transform(X_test[[\"engine_size_cc\"]])\n",
    "print(f\"Poly features: {poly.get_feature_names_out()}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Replace engine_size_cc with engine + engine²\n",
    "X_train_poly = X_train.drop(columns=[\"engine_size_cc\"])\n",
    "X_test_poly = X_test.drop(columns=[\"engine_size_cc\"])\n",
    "X_train_poly[\"engine_cc\"] = engine_train[:, 0]\n",
    "X_train_poly[\"engine_cc²\"] = engine_train[:, 1]\n",
    "X_test_poly[\"engine_cc\"] = engine_test[:, 0]\n",
    "X_test_poly[\"engine_cc²\"] = engine_test[:, 1]\n",
    "\n",
    "print(f\"Features: {X_train_poly.shape[1]} (added engine_cc²)\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "poly_model = LinearRegression()\n",
    "poly_model.fit(X_train_poly, y_train)\n",
    "y_pred_poly = poly_model.predict(X_test_poly)\n",
    "\n",
    "r2_poly = r2_score(y_test, y_pred_poly)\n",
    "rmse_poly = np.sqrt(mean_squared_error(y_test, y_pred_poly))\n",
    "\n",
    "print(f\"Polynomial (engine² added):\")\n",
    "print(f\"  R²:   {r2_poly:.3f}\")\n",
    "print(f\"  RMSE: ₪{rmse_poly:.1f}K\")\n",
    "print(f\"\\nImprovement from linear: R² {r2_full:.2f} → {r2_poly:.2f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding engine_size² captured the curved relationship — a modest but real improvement. The model now knows that a 2000cc engine isn't just \"twice as good\" as 1000cc; the benefit plateaus for large engines.\n",
    "\n",
    "### ⚠️ Common Mistake: More Polynomial = Better?\n",
    "\n",
    "What if we go further? Let's add polynomial features to ALL numeric columns and increase the degree."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Polynomial on ALL numeric features — degrees 1, 2, 3\n",
    "num_cols = [\"car_age_years\", \"km_driven_thousands\", \"engine_size_cc\",\n",
    "            \"num_previous_owners\", \"has_sunroof\"]\n",
    "dum_cols = [c for c in X_train.columns if c not in num_cols]\n",
    "\n",
    "print(f\"{'Degree':<8} {'Features':<10} {'R² (train)':<12} {'R² (test)':<12}\")\n",
    "print(\"-\" * 42)\n",
    "for degree in [1, 2, 3]:\n",
    "    pf = PolynomialFeatures(degree=degree, include_bias=False)\n",
    "    Xtr_p = np.hstack([pf.fit_transform(X_train[num_cols]),\n",
    "                        X_train[dum_cols].values])\n",
    "    Xte_p = np.hstack([pf.transform(X_test[num_cols]),\n",
    "                        X_test[dum_cols].values])\n",
    "    m = LinearRegression().fit(Xtr_p, y_train)\n",
    "    r2_tr = m.score(Xtr_p, y_train)\n",
    "    r2_te = r2_score(y_test, m.predict(Xte_p))\n",
    "    print(f\"{degree:<8} {Xtr_p.shape[1]:<10} {r2_tr:<12.3f} {r2_te:<12.3f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Degree 3 on all numeric features fits training data better — but test R² drops. It memorized the training data, including the noise. Classic **overfitting**.\n",
    "\n",
    "The sweet spot was our targeted approach: add engine² for the one feature that actually has a curve. Blindly adding polynomial terms to everything creates noise, not signal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 💬 Instructor Discussion: Diagnose the Overfitting\n",
    "\n",
    "> **Discuss with the class:**\n",
    ">\n",
    "> - At which degree does overfitting start? Where does the train-test gap widen dramatically?\n",
    "> - If Dana deployed the degree-3 all-features model, what would happen when a new car is listed?\n",
    "> - Why is the targeted engine² approach better? Think in business terms.\n",
    "\n",
    "<details>\n",
    "<summary>Key points</summary>\n",
    "\n",
    "- Overfitting starts at degree 3 — training R² keeps climbing but test R² drops. The gap = memorizing, not learning.\n",
    "- If Dana deployed degree-3-all, the model would give wildly wrong prices for unseen cars — sellers would lose trust instantly.\n",
    "- The targeted engine² approach works because we **know** engine_size has a curve (scatter plot). We're adding information, not noise. In business: \"Big engines have diminishing returns on price — buyers don't pay proportionally more for 3.5L vs 2.0L.\"\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 💡 Insight\n",
    "\n",
    "In business, a model that's \"almost perfect\" on historical data but crashes on new data is worse than a simpler model that's consistently decent. Dana wants reliability, not perfection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 🛡️ Section 5: Regularization — Ridge & Lasso\n",
    "*Keeping coefficients under control. Lasso even drops useless features automatically.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📖 Why Regularization?\n",
    "\n",
    "The polynomial overfitting happened because coefficients grew huge to fit every training point. Regularization adds a penalty: the model must fit the data AND keep coefficients small.\n",
    "\n",
    "- **Ridge (L2)**: shrinks all coefficients toward zero — none disappear completely\n",
    "- **Lasso (L1)**: can shrink some coefficients all the way to zero — automatic feature selection\n",
    "\n",
    "**Important**: Ridge and Lasso penalize coefficients by size. If features have different scales (engine_size_cc is 1000–3500 while has_sunroof is 0–1), the penalty hits them unequally. We must **standardize** first so the penalty is fair.\n",
    "\n",
    "> **A note on scaling**: In practice, standardization (e.g. `StandardScaler`) is recommended even for plain linear regression — it helps with numerical stability, convergence, and makes coefficients comparable across features. We skipped it in Sections 2–4 deliberately so you could read coefficients in their original units (₪K per year, ₪K per 1000 km, etc.) and build intuition. From here on we scale properly."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Scale features so regularization penalty is fair across all features\n",
    "scaler = StandardScaler()\n",
    "X_train_sc = pd.DataFrame(scaler.fit_transform(X_train),\n",
    "                          columns=X_train.columns, index=X_train.index)\n",
    "X_test_sc = pd.DataFrame(scaler.transform(X_test),\n",
    "                         columns=X_test.columns, index=X_test.index)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "ridge_model = Ridge(alpha=1.0)\n",
    "ridge_model.fit(X_train_sc, y_train)\n",
    "y_pred_ridge = ridge_model.predict(X_test_sc)\n",
    "\n",
    "r2_ridge = r2_score(y_test, y_pred_ridge)\n",
    "rmse_ridge = np.sqrt(mean_squared_error(y_test, y_pred_ridge))\n",
    "\n",
    "print(f\"Ridge (α=1.0):\")\n",
    "print(f\"  R²:   {r2_ridge:.3f}\")\n",
    "print(f\"  RMSE: ₪{rmse_ridge:.1f}K\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "lasso_model = Lasso(alpha=0.5)\n",
    "lasso_model.fit(X_train_sc, y_train)\n",
    "y_pred_lasso = lasso_model.predict(X_test_sc)\n",
    "\n",
    "r2_lasso = r2_score(y_test, y_pred_lasso)\n",
    "rmse_lasso = np.sqrt(mean_squared_error(y_test, y_pred_lasso))\n",
    "\n",
    "print(f\"Lasso (α=0.5):\")\n",
    "print(f\"  R²:   {r2_lasso:.3f}\")\n",
    "print(f\"  RMSE: ₪{rmse_lasso:.1f}K\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "lasso_coefs = pd.Series(lasso_model.coef_, index=X_train_sc.columns)\n",
    "\n",
    "# Features Lasso kept (non-zero)\n",
    "kept = lasso_coefs[lasso_coefs != 0].sort_values(key=abs, ascending=False)\n",
    "dropped = lasso_coefs[lasso_coefs == 0]\n",
    "\n",
    "print(f\"Lasso KEPT {len(kept)} features, DROPPED {len(dropped)}:\\n\")\n",
    "print(\"Kept (coefficients on standardized features):\")\n",
    "for feat, val in kept.items():\n",
    "    print(f\"  {feat:30s}: {val:+.2f}\")\n",
    "print(f\"\\nDropped (coefficient = 0):\")\n",
    "for feat in dropped.index:\n",
    "    print(f\"  {feat}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📖 Why Lasso Drops Features\n",
    "\n",
    "Lasso decided that `has_sunroof` and some other features don't add enough predictive value to justify their coefficient. It zeroed them out — effectively saying \"these don't matter once you know the important features.\" This is automatic feature selection, and it makes the model simpler and more interpretable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🎯 Exercise 2: Ridge vs. Lasso Coefficients\n",
    "\n",
    "- Compare the `car_age_years` and `km_driven_thousands` coefficients for Ridge and Lasso (both trained on standardized features, so they're directly comparable)\n",
    "- Which model shrinks coefficients more aggressively?\n",
    "- Why does regularization help with multicollinearity?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "##### Your code goes here #####\n",
    "\n",
    "# Print car_age and km_driven coefficients for Ridge and Lasso\n",
    "# Hint: ridge_model.coef_, lasso_coefs (both on standardized features)\n",
    "\n",
    "# Which model has more stable (smaller) coefficients?\n",
    "\n",
    "# Why does regularization help with multicollinearity? (comment)\n",
    "\n",
    "###### End of Your Code ######"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ✅ Solution\n",
    "\n",
    "<details>\n",
    "<summary>Click to reveal</summary>\n",
    "\n",
    "```python\n",
    "ridge_coefs = pd.Series(ridge_model.coef_, index=X_train_sc.columns)\n",
    "\n",
    "print(f\"{'Model':<15} {'car_age':>10} {'km_driven':>12}\")\n",
    "print(\"-\" * 37)\n",
    "print(f\"{'Ridge':<15} {ridge_coefs['car_age_years']:>10.2f} {ridge_coefs['km_driven_thousands']:>12.2f}\")\n",
    "print(f\"{'Lasso':<15} {lasso_coefs['car_age_years']:>10.2f} {lasso_coefs['km_driven_thousands']:>12.2f}\")\n",
    "\n",
    "# Because we standardized, these coefficients are directly comparable.\n",
    "# Ridge shrinks both, distributing the shared effect more evenly.\n",
    "# Without regularization, correlated features \"fight for credit\" — coefficients\n",
    "# can bounce wildly. Regularization resolves the ambiguity by penalizing large values.\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 💡 Insight\n",
    "\n",
    "Regularization isn't just about overfitting — it also stabilizes models with correlated features. For Dana, this means the pricing engine gives consistent results even if the training data shifts slightly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 🔁 Section 6: Cross-Validation\n",
    "*Don't trust one exam. Take five.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📖 Why Cross-Validation?\n",
    "\n",
    "Every score we've reported depends on which 20% ended up in the test set. A different random split could give different numbers. Cross-validation trains and tests on multiple splits, averaging the results for a more reliable estimate."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# 5-fold CV on the full linear model\n",
    "cv_scores = cross_val_score(LinearRegression(), X, y, cv=5, scoring=\"r2\")\n",
    "\n",
    "print(\"5-Fold Cross-Validation (Linear Regression):\")\n",
    "print(f\"  R² per fold: {cv_scores.round(3)}\")\n",
    "print(f\"  Mean R²:     {cv_scores.mean():.3f}\")\n",
    "print(f\"  Std:         {cv_scores.std():.3f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "models = {\n",
    "    \"Linear Regression\": LinearRegression(),\n",
    "    \"Ridge (α=1.0)\": make_pipeline(StandardScaler(), Ridge(alpha=1.0)),\n",
    "    \"Lasso (α=0.5)\": make_pipeline(StandardScaler(), Lasso(alpha=0.5)),\n",
    "}\n",
    "\n",
    "print(f\"{'Model':<25} {'Mean R²':<12} {'Std':<10}\")\n",
    "print(\"-\" * 47)\n",
    "for name, model in models.items():\n",
    "    scores = cross_val_score(model, X, y, cv=5, scoring=\"r2\")\n",
    "    print(f\"{name:<25} {scores.mean():<12.3f} {scores.std():<10.3f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 💬 Instructor Discussion: Why Cross-Validation Matters\n",
    "\n",
    "> **Discuss with the class:**\n",
    ">\n",
    "> - Compare `r2_full` (single split) with the CV mean R². Are they close?\n",
    "> - Which number do you trust more? Why?\n",
    "> - If you were reporting to Dana, which would you use?\n",
    "\n",
    "<details>\n",
    "<summary>Key points</summary>\n",
    "\n",
    "- The single split might be slightly higher or lower than CV — that's the point. CV averages over randomness, giving a more reliable estimate.\n",
    "- Report to Dana: \"Our model has R² ≈ 0.76 ± 0.03\" — the ± shows reliability. A single number without uncertainty is misleading.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 🏆 Section 7: The Grand Comparison\n",
    "*Five models. One recommendation for Dana.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "results = []\n",
    "model_preds = {\n",
    "    \"Simple (age only)\": y_pred_simple,\n",
    "    \"Multiple (all features)\": y_pred_full,\n",
    "    \"Poly (engine²)\": y_pred_poly,\n",
    "    \"Ridge (α=1.0)\": y_pred_ridge,\n",
    "    \"Lasso (α=0.5)\": y_pred_lasso,\n",
    "}\n",
    "\n",
    "for name, preds in model_preds.items():\n",
    "    results.append({\n",
    "        \"Model\": name,\n",
    "        \"R²\": r2_score(y_test, preds),\n",
    "        \"RMSE (₪K)\": np.sqrt(mean_squared_error(y_test, preds)),\n",
    "        \"MAE (₪K)\": mean_absolute_error(y_test, preds),\n",
    "    })\n",
    "\n",
    "results_df = pd.DataFrame(results).set_index(\"Model\")\n",
    "results_df = results_df.round(2)\n",
    "print(results_df.to_string())"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "results_df[\"R²\"].plot(kind=\"barh\", figsize=(8, 4), color=\"#214073\", legend=False)\n",
    "plt.title(\"Model Comparison — R² Score\")\n",
    "plt.xlabel(\"R²\")\n",
    "plt.xlim(0, 1)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 💬 Instructor Discussion: Which Model for Dana?\n",
    "\n",
    "> **Part 1 — Model selection:**\n",
    "> Dana needs **accuracy** (reasonable RMSE) and **interpretability** (explain to sellers WHY the price is what it is).\n",
    "> Which models perform best on accuracy? Which are also interpretable? What's your recommendation?\n",
    ">\n",
    "> **Part 2 — 3-minute CEO pitch:**\n",
    "> 1. **The model**: Which model and why? State the key metric (R², RMSE).\n",
    "> 2. **Top factors**: What drives used car prices most? Top 3 with ₪ impact.\n",
    "> 3. **A surprise**: What counterintuitive pattern did the data reveal?\n",
    "> 4. **Limitation**: What can't the model predict? What should Dana collect next?\n",
    "\n",
    "<details>\n",
    "<summary>Key points</summary>\n",
    "\n",
    "**Model selection:**\n",
    "- Multiple LR, Poly (engine²), Ridge, and Lasso all perform well. Simple LR is clearly not enough.\n",
    "- **Multiple LR and Ridge** are most interpretable — each feature has a clear \"price tag\" coefficient Dana can show sellers. Lasso is good if Dana wants fewer features.\n",
    "- **Recommendation: Ridge for production** — stable, accurate, interpretable, handles multicollinearity gracefully. Lasso as secondary analysis to identify which features truly matter.\n",
    "\n",
    "**CEO pitch:**\n",
    "1. **Ridge regression** — reliable and interpretable. CV R² ≈ 0.76, meaning it explains ~76% of price variation consistently.\n",
    "2. **Top factors**: Brand (BMW premium: other brands ₪20-45K cheaper), fuel type (Electric +₪30K), car age (−₪5-10K/year).\n",
    "3. **Surprise**: `has_sunroof` barely matters — Lasso dropped it entirely. Sellers who think sunroofs add major value are wrong.\n",
    "4. **Limitations**: Can't see condition, accident history, or photos. Adding a condition rating and photo quality score could improve predictions substantially.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 🎓 What You Just Did\n",
    "\n",
    "- Loaded 350 used-car listings and explored price drivers with EDA\n",
    "- Built simple regression (1 feature) and multiple regression (all features)\n",
    "- Added a targeted polynomial feature (engine²) and diagnosed overfitting at higher degrees\n",
    "- Applied Ridge and Lasso regularization — and watched Lasso drop useless features\n",
    "- Used 5-fold cross-validation for reliable model scoring\n",
    "- Compared 5 models and recommended one for a real business use case\n",
    "\n",
    "### The Regression Toolkit\n",
    "\n",
    "```python\n",
    "LinearRegression()                    # straight lines\n",
    "PolynomialFeatures(degree=2)          # add curves (targeted)\n",
    "Ridge(alpha=1.0)                      # stable coefficients\n",
    "Lasso(alpha=0.5)                      # feature selection\n",
    "cross_val_score(model, X, y, cv=5)    # reliable evaluation\n",
    "```\n",
    "\n",
    "All follow the same API: `create → .fit() → .predict()`. Same pattern as classification. Muscle memory.\n",
    "\n",
    "### 📅 What's Next\n",
    "\n",
    "- **Lecture 5**: Clustering — finding hidden groups in unlabeled data (unsupervised learning)\n",
    "\n",
    "### 💬 One Last Thing\n",
    "\n",
    "Last week you classified. This week you predicted prices. You've now mastered both flavors of supervised learning. Next week: what happens when there are NO labels? The algorithm finds structure on its own. That's unsupervised learning — and it's where things get really interesting."
   ]
  }
 ]
}