{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11.0"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recitation 8: Neural Networks (MLP) with scikit-learn\n",
    "### Data Science for Entrepreneurship-Economy | Spring 2026\n",
    "\n",
    "**Today's goal**: Build your first neural network and compare it against the traditional classifiers from Recitation 3.\n",
    "\n",
    "By the end of this session you'll be able to:\n",
    "- Train an MLP classifier using scikit-learn's familiar API\n",
    "- Choose hidden layer sizes and activation functions\n",
    "- Diagnose overfitting in neural networks\n",
    "- Compare MLP vs. Decision Tree / Logistic Regression\n",
    "- Decide when a neural network is worth the complexity\n",
    "\n",
    "**Time**: ~90 minutes | **Exercises**: 3 | **Discussions**: 3 | **Format**: Guided walkthrough\n",
    "\n",
    "**The big question**: Lecture 8 showed that MLPs learn feature interactions automatically. Today we test that claim with real data.\n",
    "\n",
    "**A note on tools**: We use scikit-learn's `MLPClassifier`, which follows the same `create \u2192 .fit() \u2192 .predict()` API you already know. sklearn's MLP runs on CPU only and offers basic architectures \u2014 for production deep learning on large datasets, frameworks like TensorFlow/Keras or PyTorch provide GPU acceleration and much more flexibility. Those are beyond this course's scope, but the concepts you learn today transfer directly."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neural_network import MLPClassifier       # the star of today's show\n",
    "from sklearn.metrics import (accuracy_score, classification_report,\n",
    "                             confusion_matrix, ConfusionMatrixDisplay)\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
    "\n",
    "sns.set_style(\"whitegrid\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "df = pd.read_csv(\"spotpark_parking.csv\")\n",
    "print(f\"Dataset: {df.shape[0]} observations \u00d7 {df.shape[1]} columns\")\n",
    "print(f\"Target balance: {df['zone_status'].value_counts().to_dict()}\")\n",
    "print(f\"  \u2192 {df['zone_status'].mean():.1%} full\\n\")\n",
    "df.head()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **SpotPark** is an Israeli smart-parking startup operating in Tel Aviv. The app helps drivers find available street parking by predicting whether a given parking zone will have free spots at a requested time. Currently, drivers waste an average of 18 minutes circling for parking \u2014 costing time, fuel, and emissions. SpotPark's data scientist, Rotem, has collected 5,000 historical records of parking zone status (available / full) along with contextual features. She wants a model that predicts parking availability so the app can route drivers toward zones with open spots.\n",
    ">\n",
    "> **Rotem's challenge**: predict parking availability from contextual features. The catch \u2014 feature INTERACTIONS drive parking patterns more than individual features."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Show why interactions matter\n",
    "fig, axes = plt.subplots(1, 2, figsize=(13, 5))\n",
    "\n",
    "# Left: restaurant count \u00d7 hour\n",
    "pivot = df.pivot_table(values=\"zone_status\", index=\"hour\",\n",
    "                       columns=pd.cut(df[\"nearby_restaurants\"], bins=[-1, 5, 15, 30],\n",
    "                                       labels=[\"Few (0-5)\", \"Some (6-15)\", \"Many (16-30)\"]),\n",
    "                       aggfunc=\"mean\")\n",
    "pivot.plot(ax=axes[0], marker=\"o\")\n",
    "axes[0].set_ylabel(\"Proportion Full\")\n",
    "axes[0].set_title(\"Restaurants \u00d7 Hour \u2192 Parking Status\")\n",
    "axes[0].legend(title=\"Restaurants nearby\")\n",
    "\n",
    "# Right: rain \u00d7 weekend\n",
    "rain_weekend = df.groupby([\"rain\", \"is_weekend\"])[\"zone_status\"].mean().unstack()\n",
    "rain_weekend.plot(kind=\"bar\", ax=axes[1], color=[\"#214073\", \"#E6592D\"])\n",
    "axes[1].set_title(\"Rain \u00d7 Weekend \u2192 Parking Status\")\n",
    "axes[1].set_xticklabels([\"No Rain\", \"Rain\"], rotation=0)\n",
    "axes[1].legend([\"Weekday\", \"Weekend\"])\n",
    "axes[1].set_ylabel(\"Proportion Full\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83d\udcd6 The Interaction Story\n",
    "\n",
    "Look at the left chart: zones with many restaurants are available in the morning but packed in the evening. A linear model sees \"more restaurants \u2192 more full\" \u2014 but the TRUTH is \"more restaurants + evening \u2192 full; more restaurants + morning \u2192 fine.\" That's an **interaction**. The MLP should capture it."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "X = df.drop(columns=[\"zone_status\", \"observation_id\"])\n",
    "y = df[\"zone_status\"]\n",
    "\n",
    "# Scale features (important for MLP \u2014 same reason as KNN in Rec 3)\n",
    "scaler = StandardScaler()\n",
    "X_scaled = pd.DataFrame(scaler.fit_transform(X), columns=X.columns, index=X.index)\n",
    "\n",
    "# Train/test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X_scaled, y, test_size=0.2, random_state=42, stratify=y\n",
    ")\n",
    "print(f\"Train: {X_train.shape[0]} | Test: {X_test.shape[0]}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Build the same baselines from Recitation 3\n",
    "baselines = {\n",
    "    \"Logistic Regression\": LogisticRegression(max_iter=1000, random_state=42),\n",
    "    \"Decision Tree (d=5)\": DecisionTreeClassifier(max_depth=5, random_state=42),\n",
    "}\n",
    "\n",
    "print(f\"{'Model':<25} {'Accuracy':<12} {'F1 (Full)':<12}\")\n",
    "print(\"-\" * 49)\n",
    "baseline_results = {}\n",
    "for name, model in baselines.items():\n",
    "    model.fit(X_train, y_train)\n",
    "    y_pred = model.predict(X_test)\n",
    "    acc = accuracy_score(y_test, y_pred)\n",
    "    report = classification_report(y_test, y_pred, output_dict=True)\n",
    "    f1_full = report[\"1\"][\"f1-score\"]\n",
    "    baseline_results[name] = {\"accuracy\": acc, \"f1_full\": f1_full, \"y_pred\": y_pred}\n",
    "    print(f\"{name:<25} {acc:<12.3f} {f1_full:<12.3f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83d\udca1 Insight\n",
    "\n",
    "Decision Tree is the best traditional model at ~77% accuracy. Not bad \u2014 but Rotem knows the interactions in her data are complex. Can an MLP do better by learning those interactions automatically?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## \ud83e\udde0 Section 2: Your First MLP\n",
    "*Same API. Same `.fit()`. Same `.predict()`. But inside \u2014 neurons, layers, and learned features.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83d\udcd6 Recap\n",
    "\n",
    "In Lecture 8 you learned that an MLP has: an **input layer** (your features), one or more **hidden layers** (where it learns interactions), and an **output layer** (the prediction). Each neuron computes a weighted sum + activation function. Training adjusts the weights to minimize prediction error.\n",
    "\n",
    "sklearn's `MLPClassifier` wraps all of this in the familiar API."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# MLP with 1 hidden layer of 32 neurons\n",
    "mlp_simple = MLPClassifier(\n",
    "    hidden_layer_sizes=(32,),        # one hidden layer, 32 neurons\n",
    "    activation=\"relu\",                # ReLU for hidden layers (from Lecture 8)\n",
    "    max_iter=500,                     # maximum training epochs\n",
    "    random_state=42,\n",
    ")\n",
    "mlp_simple.fit(X_train, y_train)\n",
    "y_pred_mlp = mlp_simple.predict(X_test)\n",
    "\n",
    "acc_mlp = accuracy_score(y_test, y_pred_mlp)\n",
    "print(f\"MLP (32 neurons, 1 layer):\")\n",
    "print(f\"  Accuracy: {acc_mlp:.3f}\")\n",
    "print(f\"  vs. Decision Tree: {baseline_results['Decision Tree (d=5)']['accuracy']:.3f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "print(classification_report(y_test, y_pred_mlp,\n",
    "      target_names=[\"Available\", \"Full\"]))"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "print(f\"Architecture: {X_train.shape[1]} inputs \u2192 32 hidden \u2192 2 outputs\")\n",
    "print(f\"Number of weight matrices: {len(mlp_simple.coefs_)}\")\n",
    "print(f\"  Layer 1 weights shape: {mlp_simple.coefs_[0].shape}\")\n",
    "print(f\"  Layer 2 weights shape: {mlp_simple.coefs_[1].shape}\")\n",
    "total_params = sum(w.size for w in mlp_simple.coefs_) + sum(b.size for b in mlp_simple.intercepts_)\n",
    "print(f\"Total parameters: {total_params}\")\n",
    "print(f\"\\nTraining iterations: {mlp_simple.n_iter_}\")\n",
    "print(f\"Final loss: {mlp_simple.loss_:.4f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple MLP with 32 hidden neurons has ~350 parameters (weights + biases). Compare: a Logistic Regression has ~9 parameters (one per feature). More parameters = more capacity to learn interactions \u2014 but also more risk of overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \u26a0\ufe0f Common Mistake\n",
    "\n",
    "If you forget to scale features before MLP, performance drops dramatically. Neural networks are sensitive to feature scales \u2014 just like KNN in Recitation 3. `StandardScaler` is non-negotiable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83c\udfaf Exercise 1: Understand the Architecture\n",
    "\n",
    "The MLP has `hidden_layer_sizes=(32,)`. That means 1 hidden layer with 32 neurons.\n",
    "\n",
    "- How many weight connections are there between the input (8 features) and the hidden layer? *(Hint: every input connects to every hidden neuron)*\n",
    "- How many between the hidden layer (32 neurons) and the output (2 classes)?\n",
    "- Does the total match `mlp_simple.coefs_` shapes from above?\n",
    "- Why does more parameters = more capacity to learn? *(Think back to Lecture 8's analogy)*"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "##### Your code goes here #####\n",
    "\n",
    "# Input \u2192 Hidden connections: 8 \u00d7 32 = ?\n",
    "# Hidden \u2192 Output connections: 32 \u00d7 2 = ?\n",
    "# Total weight parameters: ? (don't forget biases: 32 + 2)\n",
    "\n",
    "# Why more parameters = more capacity? (comment)\n",
    "\n",
    "###### End of Your Code ######"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>\u2705 Solution</summary>\n",
    "\n",
    "```python\n",
    "# Input \u2192 Hidden: 8 \u00d7 32 = 256\n",
    "# Hidden \u2192 Output: 32 \u00d7 2 = 64\n",
    "# Total weights: 256 + 64 = 320\n",
    "# Plus biases: 32 + 2 = 34\n",
    "# Grand total: 320 + 34 = 354 parameters\n",
    "\n",
    "print(f\"Input \u2192 Hidden: {8 * 32}\")\n",
    "print(f\"Hidden \u2192 Output: {32 * 2}\")\n",
    "print(f\"Biases: {32 + 2}\")\n",
    "print(f\"Total: {8 * 32 + 32 * 2 + 32 + 2}\")\n",
    "```\n",
    "\n",
    "More parameters means the network can represent more complex decision boundaries \u2014 it has more \"knobs to tune.\" But more knobs also means it can memorize noise (overfitting) if data is insufficient.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## \ud83c\udfd7\ufe0f Section 3: Architecture Exploration\n",
    "*How many layers? How many neurons? The architecture IS the design decision.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83d\udcd6 Recap\n",
    "\n",
    "In Lecture 8 you learned that hidden layers create new features: layer 1 might learn \"rain + evening,\" layer 2 might combine that with \"many restaurants.\" More layers = more abstract combinations. But more layers also means more parameters and more overfitting risk. Let's test different architectures."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "architectures = {\n",
    "    \"(16,)\":         (16,),\n",
    "    \"(32,)\":         (32,),\n",
    "    \"(64,)\":         (64,),\n",
    "    \"(32, 16)\":      (32, 16),\n",
    "    \"(64, 32)\":      (64, 32),\n",
    "    \"(64, 32, 16)\":  (64, 32, 16),\n",
    "}\n",
    "\n",
    "print(f\"{'Architecture':<20} {'Accuracy':<12} {'F1 (Full)':<12} {'Parameters':<12}\")\n",
    "print(\"-\" * 56)\n",
    "\n",
    "arch_results = {}\n",
    "for name, layers in architectures.items():\n",
    "    mlp = MLPClassifier(hidden_layer_sizes=layers, activation=\"relu\",\n",
    "                        max_iter=500, random_state=42)\n",
    "    mlp.fit(X_train, y_train)\n",
    "    y_p = mlp.predict(X_test)\n",
    "    acc = accuracy_score(y_test, y_p)\n",
    "    report = classification_report(y_test, y_p, output_dict=True)\n",
    "    f1 = report[\"1\"][\"f1-score\"]\n",
    "    n_params = sum(w.size for w in mlp.coefs_) + sum(b.size for b in mlp.intercepts_)\n",
    "    arch_results[name] = {\"accuracy\": acc, \"f1_full\": f1, \"n_params\": n_params, \"y_pred\": y_p}\n",
    "    print(f\"{name:<20} {acc:<12.3f} {f1:<12.3f} {n_params:<12d}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "names = list(arch_results.keys())\n",
    "accs = [arch_results[n][\"accuracy\"] for n in names]\n",
    "params = [arch_results[n][\"n_params\"] for n in names]\n",
    "\n",
    "fig, ax1 = plt.subplots(figsize=(10, 5))\n",
    "x = range(len(names))\n",
    "ax1.bar(x, accs, color=\"#214073\", alpha=0.7, label=\"Accuracy\")\n",
    "ax1.set_ylabel(\"Accuracy\")\n",
    "ax1.set_ylim(0.65, 0.90)\n",
    "ax1.set_xticks(x)\n",
    "ax1.set_xticklabels(names, rotation=30)\n",
    "\n",
    "ax2 = ax1.twinx()\n",
    "ax2.plot(x, params, \"o-\", color=\"#E6592D\", linewidth=2, label=\"Parameters\")\n",
    "ax2.set_ylabel(\"Number of Parameters\")\n",
    "\n",
    "ax1.legend(loc=\"upper left\")\n",
    "ax2.legend(loc=\"upper right\")\n",
    "plt.title(\"Architecture Size vs. Accuracy \u2014 Diminishing Returns\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the diminishing returns: going from (16,) to (32,16) improves accuracy noticeably. But going from (32,16) to (64,32,16) barely helps \u2014 while tripling the parameter count. More complexity doesn't always mean better predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83d\udcac Discussion 1: Pick the Architecture *(Instructor-Led)*\n",
    "\n",
    "Look at the table and chart above. Let's discuss as a class:\n",
    "- Which architecture gives the best accuracy?\n",
    "- Which gives the best accuracy-per-parameter (best performance with fewest parameters)?\n",
    "- Rotem's app runs on users' phones with limited compute. Which architecture would you recommend and why?\n",
    "- Is the 3-layer architecture worth the extra complexity? Why or why not?\n",
    "\n",
    "<details><summary>\u2705 Discussion Points</summary>\n",
    "\n",
    "(32, 16) likely gives the best accuracy-per-parameter tradeoff. For mobile: (32, 16) \u2014 small enough for phone inference, accurate enough to be useful. The 3-layer network adds ~3x parameters for marginal gain \u2014 classic diminishing returns.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83d\udca1 Insight\n",
    "\n",
    "Architecture design is the ONE thing the human still controls in neural networks. The model learns the weights \u2014 but YOU choose how many layers and neurons. With sklearn's `MLPClassifier`, this is just a tuple: `(32, 16)` means 2 layers with 32 and 16 neurons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## \u26a0\ufe0f Section 4: Overfitting \u2014 When the Network Memorizes\n",
    "*More capacity without more data = trouble. Neural networks overfit harder than anything you've seen.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Deliberately overfit: huge network, many iterations\n",
    "mlp_overfit = MLPClassifier(\n",
    "    hidden_layer_sizes=(256, 128, 64),   # massive for our dataset\n",
    "    activation=\"relu\",\n",
    "    max_iter=2000,\n",
    "    random_state=42,\n",
    ")\n",
    "mlp_overfit.fit(X_train, y_train)\n",
    "\n",
    "train_acc = mlp_overfit.score(X_train, y_train)\n",
    "test_acc = mlp_overfit.score(X_test, y_test)\n",
    "\n",
    "print(f\"Overfit MLP (256, 128, 64):\")\n",
    "print(f\"  Train accuracy: {train_acc:.3f}\")\n",
    "print(f\"  Test accuracy:  {test_acc:.3f}\")\n",
    "print(f\"  Gap:            {train_acc - test_acc:.3f}\")\n",
    "n_params = sum(w.size for w in mlp_overfit.coefs_) + sum(b.size for b in mlp_overfit.intercepts_)\n",
    "print(f\"  Parameters:     {n_params:,}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# sklearn stores the loss at each iteration\n",
    "plt.figure(figsize=(10, 5))\n",
    "plt.plot(mlp_overfit.loss_curve_, color=\"#E6592D\", linewidth=2)\n",
    "plt.xlabel(\"Iteration (Epoch)\")\n",
    "plt.ylabel(\"Training Loss\")\n",
    "plt.title(\"Training Loss Curve \u2014 Overfit MLP\")\n",
    "plt.grid(True, alpha=0.3)\n",
    "plt.show()\n",
    "print(\"Training loss keeps dropping \u2014 but test accuracy doesn't improve.\")\n",
    "print(\"The model is memorizing the training data, not learning general patterns.\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \u26a0\ufe0f Common Mistake\n",
    "\n",
    "A high training accuracy with a much lower test accuracy is classic **overfitting**. The large (256, 128, 64) network has tens of thousands of parameters for only 4,000 training samples. The model has enough capacity to memorize every training example, including the noise."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# early_stopping=True monitors validation loss and stops when it starts rising\n",
    "mlp_earlystop = MLPClassifier(\n",
    "    hidden_layer_sizes=(64, 32),\n",
    "    activation=\"relu\",\n",
    "    max_iter=2000,\n",
    "    early_stopping=True,          # the fix\n",
    "    validation_fraction=0.15,     # use 15% of training data for validation\n",
    "    n_iter_no_change=20,          # stop if no improvement for 20 iterations\n",
    "    random_state=42,\n",
    ")\n",
    "mlp_earlystop.fit(X_train, y_train)\n",
    "\n",
    "print(f\"MLP with early stopping:\")\n",
    "print(f\"  Stopped at iteration: {mlp_earlystop.n_iter_}\")\n",
    "print(f\"  Train accuracy: {mlp_earlystop.score(X_train, y_train):.3f}\")\n",
    "print(f\"  Test accuracy:  {mlp_earlystop.score(X_test, y_test):.3f}\")"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83c\udfaf Exercise 2: Diagnose Overfitting\n",
    "\n",
    "Compare three models: the overfit MLP (256,128,64), the early-stopped MLP (64,32), and the simple MLP (32,) from Section 2.\n",
    "- For each, print train accuracy AND test accuracy\n",
    "- Which has the smallest train-test gap?\n",
    "- Which has the best TEST accuracy? (Test is what matters \u2014 not train)\n",
    "- What's the lesson about model size vs. dataset size?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "##### Your code goes here #####\n",
    "\n",
    "# Print train + test accuracy for all three\n",
    "\n",
    "# Smallest gap:\n",
    "# Best test accuracy:\n",
    "# Lesson about size vs data: (comment)\n",
    "\n",
    "###### End of Your Code ######"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>\u2705 Solution</summary>\n",
    "\n",
    "```python\n",
    "models_check = {\n",
    "    \"MLP (32,) simple\": mlp_simple,\n",
    "    \"MLP (64,32) early stop\": mlp_earlystop,\n",
    "    \"MLP (256,128,64) overfit\": mlp_overfit,\n",
    "}\n",
    "\n",
    "print(f\"{'Model':<30} {'Train Acc':<12} {'Test Acc':<12} {'Gap':<10}\")\n",
    "print(\"-\" * 64)\n",
    "for name, m in models_check.items():\n",
    "    tr = m.score(X_train, y_train)\n",
    "    te = m.score(X_test, y_test)\n",
    "    print(f\"{name:<30} {tr:<12.3f} {te:<12.3f} {tr - te:<10.3f}\")\n",
    "```\n",
    "\n",
    "The simple or early-stopped MLP likely has the best test accuracy. The overfit MLP has the highest train accuracy but worst test accuracy. Lesson: more parameters need more data. Early stopping is your seatbelt.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83d\udca1 Insight\n",
    "\n",
    "Neural networks are powerful but greedy \u2014 they'll eat all the data AND the noise if you let them. Early stopping is your seatbelt. For Rotem's dataset, a modest architecture with early stopping is the responsible choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## \ud83c\udfc6 Section 5: The Grand Comparison\n",
    "*MLP vs. the Recitation 3 classics. Same data, same evaluation, different models.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "# Use the best architecture from Section 3\n",
    "mlp_best = MLPClassifier(hidden_layer_sizes=(32, 16), activation=\"relu\",\n",
    "                          max_iter=500, random_state=42)\n",
    "mlp_best.fit(X_train, y_train)\n",
    "y_pred_best_mlp = mlp_best.predict(X_test)\n",
    "report_mlp = classification_report(y_test, y_pred_best_mlp, output_dict=True)\n",
    "\n",
    "all_results = []\n",
    "for name, res in baseline_results.items():\n",
    "    all_results.append({\"Model\": name, \"Accuracy\": res[\"accuracy\"], \"F1 (Full)\": res[\"f1_full\"]})\n",
    "\n",
    "all_results.append({\n",
    "    \"Model\": \"MLP (32, 16)\",\n",
    "    \"Accuracy\": accuracy_score(y_test, y_pred_best_mlp),\n",
    "    \"F1 (Full)\": report_mlp[\"1\"][\"f1-score\"],\n",
    "})\n",
    "\n",
    "results_df = pd.DataFrame(all_results).set_index(\"Model\").round(3)\n",
    "results_df"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "results_df.plot(kind=\"barh\", figsize=(10, 4), color=[\"#214073\", \"#E6592D\"])\n",
    "plt.title(\"Model Comparison \u2014 Parking Zone Classification\")\n",
    "plt.xlabel(\"Score\")\n",
    "plt.xlim(0.5, 0.90)\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n",
    "models_compare = [\n",
    "    (\"Logistic Reg.\", baseline_results[\"Logistic Regression\"][\"y_pred\"]),\n",
    "    (\"Decision Tree\", baseline_results[\"Decision Tree (d=5)\"][\"y_pred\"]),\n",
    "    (\"MLP (32, 16)\", y_pred_best_mlp),\n",
    "]\n",
    "\n",
    "for ax, (name, preds) in zip(axes, models_compare):\n",
    "    cm = confusion_matrix(y_test, preds)\n",
    "    ConfusionMatrixDisplay(cm, display_labels=[\"Available\", \"Full\"]).plot(ax=ax, cmap=\"Blues\")\n",
    "    ax.set_title(name, fontsize=10)\n",
    "\n",
    "plt.suptitle(\"Confusion Matrices \u2014 All Models\", fontsize=13)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83d\udcac Discussion 2: Where Does the MLP Win? *(Instructor-Led)*\n",
    "\n",
    "Look at the comparison table and confusion matrices. Let's discuss as a class:\n",
    "- On which class (Available or Full) does the MLP improve most over Decision Tree?\n",
    "- Why might the MLP be better at predicting \"Full\"? *(Think about the feature interactions from Section 1's EDA)*\n",
    "- Is the accuracy improvement worth the added complexity of a neural network? Consider Rotem's context.\n",
    "\n",
    "<details><summary>\u2705 Discussion Points</summary>\n",
    "\n",
    "MLP likely improves most on \"Full\" \u2014 because \"Full\" is driven by complex interactions (restaurants \u00d7 evening, rain \u00d7 rush hour) that linear models miss. Each correct prediction saves a driver ~18 minutes of circling. At scale (thousands of daily predictions), the improvement is significant. Worth the complexity for a production app.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83c\udfaf Exercise 3: Cross-Validation Check\n",
    "\n",
    "Run 5-fold cross-validation on the best MLP architecture and on Decision Tree.\n",
    "- Compare the mean \u00b1 std of accuracy\n",
    "- Is the MLP advantage consistent across folds, or just lucky on one split?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "source": [
    "##### Your code goes here #####\n",
    "\n",
    "# 5-fold CV for Decision Tree\n",
    "\n",
    "# 5-fold CV for MLP (best architecture)\n",
    "\n",
    "# Is the advantage consistent? (comment)\n",
    "\n",
    "###### End of Your Code ######"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>\u2705 Solution</summary>\n",
    "\n",
    "```python\n",
    "dt_cv = cross_val_score(DecisionTreeClassifier(max_depth=5, random_state=42),\n",
    "                        X_scaled, y, cv=5, scoring=\"accuracy\")\n",
    "mlp_cv = cross_val_score(MLPClassifier(hidden_layer_sizes=(32, 16), activation=\"relu\",\n",
    "                                        max_iter=500, random_state=42),\n",
    "                         X_scaled, y, cv=5, scoring=\"accuracy\")\n",
    "\n",
    "print(f\"Decision Tree:  {dt_cv.mean():.3f} \u00b1 {dt_cv.std():.3f}\")\n",
    "print(f\"MLP (32, 16):   {mlp_cv.mean():.3f} \u00b1 {mlp_cv.std():.3f}\")\n",
    "```\n",
    "\n",
    "MLP should show consistently higher accuracy across folds (with slightly higher variance). If the advantage holds across 5 folds, it's real \u2014 not luck. Note: `cross_val_score` with MLP may be slow on CPU.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## \ud83e\udd14 Section 6: When to Go Deep\n",
    "*The MLP won today. But it doesn't always win. Know when to use it.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83d\udcd6 The Decision Framework\n",
    "\n",
    "From Lecture 8, the deep learning decision checklist:\n",
    "\n",
    "1. **Did you try simpler models first?** Always start with a baseline (Logistic Regression, Decision Tree). If they're good enough, stop.\n",
    "2. **Do you have enough data?** With sklearn's CPU-only MLP, even a few thousand rows can work. Production deep learning frameworks (TensorFlow, PyTorch) with GPU can handle millions \u2014 but that's beyond our tools.\n",
    "3. **Are there complex feature interactions?** If a linear model works well, the MLP won't add much. If interactions drive the outcome (like parking), the MLP shines.\n",
    "4. **Can you tolerate a black box?** The MLP doesn't give you feature importance or interpretable rules. If Rotem needs to explain WHY a zone is predicted \"Full,\" a Decision Tree is better.\n",
    "5. **Is inference speed acceptable?** For a phone app predicting in real-time, a small MLP is fast. A huge network would be too slow without GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \ud83d\udcac Discussion 3: Rotem's Final Recommendation *(Instructor-Led)*\n",
    "\n",
    "> Rotem presents to SpotPark's CTO. Let's discuss her recommendation as a class:\n",
    ">\n",
    "> 1. **Which model to deploy** and why? (State the accuracy and F1)\n",
    "> 2. **What the MLP learned** that traditional models couldn't (reference the feature interactions)\n",
    "> 3. **The tradeoff** she accepted by choosing MLP over Decision Tree (black box, compute)\n",
    "> 4. **A fallback plan**: If the MLP breaks in production, what should the system fall back to?\n",
    "> 5. **Limitations of our approach**: What would Rotem need to use production deep learning frameworks (TensorFlow/PyTorch) instead of sklearn? (Hint: GPU, larger data, more flexible architectures)\n",
    "\n",
    "<details><summary>\u2705 Discussion Points</summary>\n",
    "\n",
    "1. **Deploy** the best MLP architecture with early stopping. Best performance of all tested models on both accuracy and F1.\n",
    "2. **Interactions**: The MLP captured restaurant \u00d7 hour interactions and rain \u00d7 weekend \u00d7 hour patterns that the Decision Tree partially missed and Logistic Regression completely missed.\n",
    "3. **Tradeoff**: MLP is a black box \u2014 Rotem can't explain \"why this zone is predicted Full\" with simple rules. A Decision Tree with its splits was more interpretable. For the app's use case (just show \"Available\" or \"Full\"), black-box is acceptable.\n",
    "4. **Fallback**: If MLP fails (e.g., convergence issues, latency), fall back to Decision Tree. It's the best traditional model and has no convergence risk.\n",
    "5. **Sklearn limitations**: sklearn's MLP runs on CPU only \u2014 with 5,000 rows this is fine (seconds to train), but with millions of rows or image/text data, training would take hours. TensorFlow/Keras or PyTorch provide GPU acceleration, support for CNNs (images) and RNNs (text), and much more flexible architectures. For SpotPark at scale, migrating to a production framework would be the next step.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### \ud83c\udf93 What You Just Did\n",
    "\n",
    "- Built your first neural network using scikit-learn's `MLPClassifier`\n",
    "- Compared MLP against Decision Tree and Logistic Regression on the same data\n",
    "- Explored architecture choices: layers, neurons, activation functions\n",
    "- Demonstrated and diagnosed overfitting \u2014 then fixed it with early stopping\n",
    "- Showed that MLP captures feature interactions that traditional models miss\n",
    "- Made an informed deploy-vs-don't-deploy recommendation\n",
    "\n",
    "### The MLP in sklearn \u2014 Three Lines\n",
    "\n",
    "```python\n",
    "mlp = MLPClassifier(hidden_layer_sizes=(32, 16), activation=\"relu\",\n",
    "                    early_stopping=True, random_state=42)\n",
    "mlp.fit(X_train, y_train)\n",
    "y_pred = mlp.predict(X_test)\n",
    "```\n",
    "\n",
    "Same `create \u2192 .fit() \u2192 .predict()` you've used since Recitation 3. The API is familiar \u2014 the power inside is new.\n",
    "\n",
    "### A Note on What Comes Next\n",
    "\n",
    "sklearn's `MLPClassifier` is a great learning tool but a limited production tool. It runs on CPU, supports only fully-connected layers, and lacks the flexibility of dedicated deep learning frameworks. For larger datasets, image recognition, text understanding, or any serious deep learning project, you'd use:\n",
    "\n",
    "- **TensorFlow / Keras** \u2014 the industry standard, Google-backed\n",
    "- **PyTorch** \u2014 research favorite, Meta-backed, very Pythonic\n",
    "- Both support GPU acceleration, convolutional layers, recurrent layers, transformers, and much more\n",
    "\n",
    "The concepts you learned today \u2014 layers, neurons, activation functions, loss, gradient descent, overfitting, early stopping \u2014 transfer directly to these frameworks. The syntax changes; the thinking doesn't.\n",
    "\n",
    "### \ud83d\udcc5 What's Next\n",
    "\n",
    "- **Final Project**: Apply everything from Lectures 1\u20138 to a real business problem\n",
    "\n",
    "### \ud83d\udcac One Last Thing\n",
    "\n",
    "\"Eight recitations ago you typed `2 + 2` in a Colab cell. Today you built a neural network that predicts parking availability by learning feature interactions no human explicitly programmed. In between, you explored data, classified customers, predicted prices, segmented subscribers, built recommendation engines, processed text and images, and now \u2014 trained a brain-inspired model. That's not a semester of exercises. That's a data science education.\""
   ]
  }
 ]
}