{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "df024652",
   "metadata": {},
   "source": [
    "# Recitation 5: Clustering & PCA with scikit-learn\n",
    "### Data Science for Entrepreneurship-Economy | Spring 2026\n",
    "\n",
    "**Today's goal**: Discover hidden customer segments in unlabeled data — from raw features to named, actionable groups visualized in 2D.\n",
    "\n",
    "By the end of this session you'll be able to:\n",
    "- Scale features and run K-Means clustering\n",
    "- Choose K using the Elbow method and Silhouette score\n",
    "- Profile and name clusters with business meaning\n",
    "- Use PCA to reduce dimensions and visualize clusters in 2D\n",
    "- Combine PCA + K-Means into a full segmentation pipeline\n",
    "\n",
    "**Time**: ~90 minutes | **Discussions**: 6 | **Exercises**: 1 | **Format**: Guided walkthrough with instructor-led discussions\n",
    "\n",
    "**The big shift**: Every model so far had a target column (churn, price, valuation). Today — NO target. The algorithm finds structure on its own."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bac6d9ea",
   "metadata": {},
   "source": [
    "### 📖 Quick Recap\n",
    "\n",
    "- In Recitations 3–4 you built classifiers and regressors — supervised learning with labeled targets\n",
    "- In Lecture 5 you learned K-Means, Silhouette scores, PCA, and cluster profiling — conceptually\n",
    "- Today you run the full unsupervised pipeline hands-on for the first time\n",
    "- Key API change: `.fit(X)` only — no `y`, no `.predict()`, no \"right answer\"\n",
    "- Scaling is mandatory (same as KNN in Recitation 3)\n",
    "\n",
    "So far, someone told the algorithm the answers. Today, the algorithm discovers them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7ad662e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "# scikit-learn imports for unsupervised learning\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.metrics import silhouette_score\n",
    "\n",
    "sns.set_style(\"whitegrid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "787bb014",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"telebari_subscribers.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d222566",
   "metadata": {},
   "source": [
    "### 📖 Meet the Business\n",
    "\n",
    "> **TeleBari** is an Israeli mobile telecom company with ~2 million subscribers. The company offers a range of plans but has never formally segmented its customer base — everyone gets the same marketing emails, the same retention offers, and the same support priority. The Head of Customer Strategy, Amit, believes the subscriber base contains distinct behavioral groups and wants data-driven segments to personalize plan recommendations, target retention campaigns, and allocate support resources. He exported a sample of 500 subscribers with usage data from the last 3 months.\n",
    "\n",
    "Amit wants segments — not guesses. Let's find them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a0f7bb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Dataset: {df.shape[0]} subscribers x {df.shape[1]} columns\\n\")\n",
    "print(df.describe().round(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdb1921c",
   "metadata": {},
   "source": [
    "Notice there's NO target column. No `churned`, no `price`, no labels. Just behavioral features. This is unsupervised learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cde37274",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(2, 4, figsize=(16, 7))\n",
    "features = [c for c in df.columns if c != \"subscriber_id\"]\n",
    "for i, feat in enumerate(features):\n",
    "    ax = axes[i // 4, i % 4]\n",
    "    df[feat].hist(bins=20, ax=ax, color=\"#214073\", edgecolor=\"white\")\n",
    "    ax.set_title(feat, fontsize=9)\n",
    "plt.tight_layout()\n",
    "plt.suptitle(\"Feature Distributions — TeleBari Subscribers\", y=1.02, fontsize=13)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00b15909",
   "metadata": {},
   "source": [
    "Key observation: `monthly_data_gb` has a much wider range than other features. If we don't scale, this one feature will dominate all distance calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60e8bdea",
   "metadata": {},
   "outputs": [],
   "source": [
    "corr = df.drop(columns=\"subscriber_id\").corr()\n",
    "sns.heatmap(corr, annot=True, fmt=\".2f\", cmap=\"coolwarm\",\n",
    "            vmin=-1, vmax=1, square=True)\n",
    "plt.title(\"Feature Correlations — TeleBari\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "138ff687",
   "metadata": {},
   "source": [
    "Notice: `monthly_minutes` and `international_minutes` are correlated — people who call a lot also call internationally. PCA will compress this shared information later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "245f9314",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop ID, keep only features\n",
    "X = df.drop(columns=\"subscriber_id\")\n",
    "\n",
    "# Scale — CRITICAL for K-Means and PCA\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "X_scaled = pd.DataFrame(X_scaled, columns=X.columns, index=X.index)\n",
    "\n",
    "print(\"Before scaling — monthly_data_gb std:\", f\"{X['monthly_data_gb'].std():.1f}\")\n",
    "print(\"After scaling  — monthly_data_gb std:\", f\"{X_scaled['monthly_data_gb'].std():.2f}\")\n",
    "print(\"\\nAll features now have mean ~ 0 and std ~ 1. Ready for clustering.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d30a31f9",
   "metadata": {},
   "source": [
    "---\n",
    "## 🎯 Section 2: K-Means Clustering\n",
    "*Let the algorithm find groups. You tell it how many — it figures out where.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "995070c0",
   "metadata": {},
   "source": [
    "### 📖 K-Means Recap\n",
    "\n",
    "K-Means partitions data into exactly K groups by repeating: assign each point to the nearest centroid → move centroids to the mean of their group → repeat until stable. You choose K. The algorithm finds the best centroids. Always on scaled data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa85dcd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)\n",
    "kmeans.fit(X_scaled)\n",
    "\n",
    "# Add cluster labels to the DataFrame\n",
    "df[\"cluster\"] = kmeans.labels_\n",
    "\n",
    "print(\"Cluster sizes:\")\n",
    "print(df[\"cluster\"].value_counts().sort_index())\n",
    "print(f\"\\nInertia: {kmeans.inertia_:.0f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d03b027",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.scatter(df[\"monthly_data_gb\"], df[\"monthly_minutes\"],\n",
    "            c=df[\"cluster\"], cmap=\"Set2\", alpha=0.6,\n",
    "            edgecolor=\"gray\", s=40)\n",
    "plt.xlabel(\"Monthly Data (GB)\")\n",
    "plt.ylabel(\"Monthly Voice (minutes)\")\n",
    "plt.title(\"Clusters: Data Usage vs. Voice Minutes\")\n",
    "plt.colorbar(label=\"Cluster\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee675252",
   "metadata": {},
   "source": [
    "Even in this 2D slice (2 of 8 features), structure is visible. The upper-left cluster (high voice, low data) looks different from the lower-right (high data, low voice). But 2D shows only part of the story — we'll use PCA later for the full picture."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cc56200",
   "metadata": {},
   "source": [
    "### ⚠️ Common Mistake\n",
    "\n",
    "We chose K=4 as a starting guess. But how do we KNOW 4 is the right number? Maybe 3 is better, or 5. Next section: let the data guide the choice."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47dd90e5",
   "metadata": {},
   "source": [
    "---\n",
    "## 🔍 Section 3: Choosing K\n",
    "*Too few groups = too vague. Too many = chaos. Find the sweet spot.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4a3b126",
   "metadata": {},
   "outputs": [],
   "source": [
    "inertias = []\n",
    "K_range = range(2, 11)\n",
    "\n",
    "for k in K_range:\n",
    "    km = KMeans(n_clusters=k, random_state=42, n_init=10)\n",
    "    km.fit(X_scaled)\n",
    "    inertias.append(km.inertia_)\n",
    "\n",
    "plt.plot(K_range, inertias, \"bo-\", linewidth=2, markersize=8)\n",
    "plt.xlabel(\"K (number of clusters)\")\n",
    "plt.ylabel(\"Inertia (within-cluster distance)\")\n",
    "plt.title(\"Elbow Method — TeleBari\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f801fc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "sil_scores = []\n",
    "for k in K_range:\n",
    "    km = KMeans(n_clusters=k, random_state=42, n_init=10)\n",
    "    labels = km.fit_predict(X_scaled)\n",
    "    sil = silhouette_score(X_scaled, labels)\n",
    "    sil_scores.append(sil)\n",
    "    print(f\"K={k}: Silhouette = {sil:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d585b70b",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 2, figsize=(13, 5))\n",
    "axes[0].plot(K_range, inertias, \"bo-\", linewidth=2)\n",
    "axes[0].set_xlabel(\"K\")\n",
    "axes[0].set_ylabel(\"Inertia\")\n",
    "axes[0].set_title(\"Elbow Method\")\n",
    "axes[0].axvline(x=4, color=\"red\", linestyle=\"--\", alpha=0.7)\n",
    "axes[1].plot(K_range, sil_scores, \"gs-\", linewidth=2)\n",
    "axes[1].set_xlabel(\"K\")\n",
    "axes[1].set_ylabel(\"Silhouette Score\")\n",
    "axes[1].set_title(\"Silhouette Score\")\n",
    "axes[1].axvline(x=4, color=\"red\", linestyle=\"--\", alpha=0.7)\n",
    "plt.suptitle(\"Both Methods Point to K=4\", fontsize=13)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30f4bf91",
   "metadata": {},
   "source": [
    "### 💬 Discussion 1: Read the Plots\n",
    "\n",
    "*Instructor leads — students respond:*\n",
    "\n",
    "- Look at the elbow plot: at which K does the curve \"bend\" (inertia stops dropping sharply)?\n",
    "- Look at the silhouette plot: which K has the highest score?\n",
    "- Do both methods agree? What K would you recommend to Amit?\n",
    "- Amit says: \"My marketing team can realistically manage 3 campaigns. Is K=3 OK?\" What do you tell him?\n",
    "\n",
    "**Takeaway**: Both methods point to **K=4** — the elbow bends at 4 and silhouette peaks at K=4. For Amit's question: K=3 is mathematically worse, but if the team can only handle 3 campaigns, merge the two most similar segments. **Business constraints override math** — but document the tradeoff. K is a business decision aided by math, not purely a math decision."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "559d1fc2",
   "metadata": {},
   "source": [
    "### 💡 Insight\n",
    "\n",
    "Math suggests K=4. The business can handle K=3 or K=4. The right answer is: try both, profile both, and let Amit choose based on which segments are most actionable. Data informs — humans decide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66d4445b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Confirmed: K=4\n",
    "kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)\n",
    "df[\"cluster\"] = kmeans.fit_predict(X_scaled)\n",
    "\n",
    "print(\"Final cluster sizes:\")\n",
    "print(df[\"cluster\"].value_counts().sort_index())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa9b24ce",
   "metadata": {},
   "source": [
    "---\n",
    "## 👥 Section 4: Cluster Profiling\n",
    "*Numbers are useless until you name them. Turn cluster 0, 1, 2, 3 into \"Data Junkies\" and \"Traditional Callers.\"*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e39be6e",
   "metadata": {},
   "source": [
    "### 📖 Profiling Recap\n",
    "\n",
    "K-Means gave us 4 groups labeled 0, 1, 2, 3. But \"0\" means nothing to Amit. Profiling computes the average feature values per cluster so we can understand WHO is in each group and WHAT makes them different. Then we name them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfff95ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute mean of each feature per cluster (on ORIGINAL scale, not scaled)\n",
    "profiles = df.groupby(\"cluster\")[X.columns.tolist()].mean().round(1)\n",
    "profiles[\"size\"] = df[\"cluster\"].value_counts().sort_index()\n",
    "print(profiles.to_string())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfbc9f3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize profiles for visualization (z-score per feature)\n",
    "prof_data = profiles.drop(columns=\"size\")\n",
    "profiles_z = (prof_data - prof_data.mean()) / prof_data.std()\n",
    "\n",
    "plt.figure(figsize=(12, 4))\n",
    "sns.heatmap(profiles_z, annot=True, fmt=\".1f\", cmap=\"RdYlGn\",\n",
    "            center=0, linewidths=1,\n",
    "            yticklabels=[f\"Cluster {i}\" for i in range(4)])\n",
    "plt.title(\"Cluster Profiles (z-scored) — Red = Below Average, Green = Above Average\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7e37031",
   "metadata": {},
   "source": [
    "### 📖 How to Read the Heatmap\n",
    "\n",
    "Each row is a cluster. Each column is a feature. Green = that cluster is ABOVE average on this feature. Red = BELOW average. The pattern of greens and reds tells you the personality of each cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f5b560c",
   "metadata": {},
   "source": [
    "### 💬 Discussion 2: Name the Segments\n",
    "\n",
    "*Instructor leads — students respond:*\n",
    "\n",
    "- Look at the profile table and heatmap\n",
    "- For each cluster (0, 1, 2, 3), suggest a **business-friendly name** based on its dominant characteristics\n",
    "- For each, describe in **one sentence** who this subscriber is\n",
    "- Hint: look for the most extreme (green/red) values in each row — those define the cluster's identity\n",
    "\n",
    "**Takeaway**: The four profiles map to:\n",
    "\n",
    "- **Data Junkies** — High data usage, high app sessions, low voice/SMS. Digital-first subscribers who stream and browse.\n",
    "- **Traditional Callers** — High voice minutes, high international, high SMS, low data, low app usage. Voice-centric, long-tenure subscribers.\n",
    "- **Budget Minimalists** — Low everything, lowest bills, short tenure, highest support calls. Deal-seekers who call support often hunting for cheaper plans.\n",
    "- **Power Users** — High across the board, highest bills, long tenure, lowest support calls. Most valuable, most satisfied subscribers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd24f3b1",
   "metadata": {},
   "source": [
    "### 💡 Insight\n",
    "\n",
    "Four clusters, four personalities, four different strategies. Data Junkies need bigger data plans. Traditional Callers need international bundles. Budget Minimalists need simple cheap plans with less support friction. Power Users need VIP treatment to keep them loyal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91206209",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(2, 4, figsize=(16, 7))\n",
    "colors = [\"#214073\", \"#008C8C\", \"#E6592D\", \"#F5A623\"]\n",
    "for i, feat in enumerate(X.columns):\n",
    "    ax = axes[i // 4, i % 4]\n",
    "    df.groupby(\"cluster\")[feat].mean().plot(\n",
    "        kind=\"bar\", ax=ax, color=colors)\n",
    "    ax.set_title(feat, fontsize=9)\n",
    "    ax.set_xlabel(\"\")\n",
    "plt.suptitle(\"Average Feature Values by Cluster\", y=1.02, fontsize=13)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ab8a43b",
   "metadata": {},
   "source": [
    "### 💬 Discussion 3: Amit's Marketing Brief\n",
    "\n",
    "*Instructor leads — students respond:*\n",
    "\n",
    "- For EACH of the 4 segments, suggest a one-line marketing action Amit should take\n",
    "- Example: \"Data Junkies → Offer unlimited data plan upgrade at ₪20/month premium\"\n",
    "- Think about: what does each group VALUE? What keeps them as a customer? What would make them switch to a competitor?\n",
    "\n",
    "**Takeaway**:\n",
    "- **Data Junkies** → Offer unlimited data plan, promote TeleBari app features, partner with streaming services\n",
    "- **Traditional Callers** → International calling bundles, family plans, avoid pushing app-only features\n",
    "- **Budget Minimalists** → Simple low-cost plans, reduce support friction with FAQ/chatbot, loyalty rewards for staying >6 months\n",
    "- **Power Users** → VIP support line, early access to new features, \"thank you\" perks — they're the most valuable, don't let a competitor poach them"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82064b22",
   "metadata": {},
   "source": [
    "---\n",
    "## 🔬 Section 5: PCA — Dimensionality Reduction\n",
    "*8 features → 2 dimensions. See all 500 subscribers on one plot.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee609086",
   "metadata": {},
   "source": [
    "### 📖 PCA Recap\n",
    "\n",
    "The 2D scatter plot in Section 2 showed only 2 of 8 features — a partial view. Some clusters that overlap in 2D may be clearly separated in the full 8D space. PCA compresses all 8 features into 2 \"super-features\" (principal components) that capture the most variance. Then we can plot EVERYTHING on one chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c45befd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA(n_components=2)\n",
    "X_pca = pca.fit_transform(X_scaled)     # fit + transform in one step\n",
    "\n",
    "print(f\"Original shape: {X_scaled.shape}\")\n",
    "print(f\"PCA shape:      {X_pca.shape}\")\n",
    "print(f\"\\nPC1 explains: {pca.explained_variance_ratio_[0]:.1%}\")\n",
    "print(f\"PC2 explains: {pca.explained_variance_ratio_[1]:.1%}\")\n",
    "print(f\"Total:        {sum(pca.explained_variance_ratio_):.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8beea175",
   "metadata": {},
   "source": [
    "### ⚠️ Common Mistake\n",
    "\n",
    "Even though 2 components capture a large share of the variance, we're still losing some information — features that don't align with PC1 or PC2 get compressed away. For ANALYSIS (not just visualization), use more components — we'll check how many we need next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dd0544a",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_full = PCA().fit(X_scaled)\n",
    "cumvar = pca_full.explained_variance_ratio_.cumsum()\n",
    "\n",
    "plt.plot(range(1, len(cumvar) + 1), cumvar, \"bo-\", linewidth=2)\n",
    "plt.xlabel(\"Number of Components\")\n",
    "plt.ylabel(\"Cumulative Variance Explained\")\n",
    "plt.axhline(y=0.80, color=\"r\", linestyle=\"--\", label=\"80% threshold\")\n",
    "plt.title(\"How Many Components Do We Need?\")\n",
    "plt.legend()\n",
    "plt.xticks(range(1, len(cumvar) + 1))\n",
    "plt.show()\n",
    "\n",
    "print(\"Cumulative variance by number of components:\")\n",
    "for i, cv in enumerate(cumvar, 1):\n",
    "    print(f\"  {i} PCs: {cv:.1%}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bf36dc6",
   "metadata": {},
   "source": [
    "2 components are enough for visualization. For deeper analysis, you'd use more — the cumulative variance plot helps you decide how many."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a2ffe05",
   "metadata": {},
   "outputs": [],
   "source": [
    "loadings = pd.DataFrame(\n",
    "    pca.components_,\n",
    "    columns=X.columns,\n",
    "    index=[\"PC1\", \"PC2\"]\n",
    ").round(2)\n",
    "\n",
    "print(\"PCA Loadings (what each component measures):\\n\")\n",
    "print(loadings.to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1c94052",
   "metadata": {},
   "source": [
    "### 💬 Discussion 4: Interpret the Components\n",
    "\n",
    "*Instructor leads — students respond:*\n",
    "\n",
    "- Look at the loadings table\n",
    "- **PC1**: Which features have the largest positive loadings? Which have the largest negative loadings? Give PC1 a business name (e.g., \"Usage Intensity\" or \"Spending Power\")\n",
    "- **PC2**: Same — which features dominate? Give PC2 a business name (e.g., \"Digital vs. Traditional\")\n",
    "- Why does this make business sense for a telecom company?\n",
    "\n",
    "**Takeaway**: **PC1** loads positively on `monthly_minutes`, `monthly_sms`, `monthly_bill_nis`, `account_age_months`, `international_minutes` and negatively on `support_calls` — it's an **\"Overall Usage / Value\"** axis. **PC2** loads positively on `monthly_data_gb` and `app_sessions_per_week` — it's a **\"Digital vs. Traditional\"** axis. The two fundamental axes of telecom behavior: how MUCH someone uses, and HOW they use it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b6ccce8",
   "metadata": {},
   "source": [
    "### 💡 Insight\n",
    "\n",
    "PCA didn't just compress numbers — it discovered the two fundamental axes of telecom customer behavior: how much they use (value) and how they use it (digital vs. traditional). These are the same dimensions Amit's team intuitively thinks about."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d22eb1c9",
   "metadata": {},
   "source": [
    "---\n",
    "## 🗺️ Section 6: The Full Picture\n",
    "*Clusters + PCA = all 500 subscribers on one chart, colored by segment.*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "697638c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 7))\n",
    "scatter = plt.scatter(X_pca[:, 0], X_pca[:, 1],\n",
    "                      c=df[\"cluster\"], cmap=\"Set2\", alpha=0.6,\n",
    "                      edgecolor=\"gray\", s=50)\n",
    "plt.xlabel(\"PC1 — Overall Usage / Value →\")\n",
    "plt.ylabel(\"PC2 — Digital ↑ vs. Traditional ↓\")\n",
    "plt.title(\"TeleBari Customer Segments in PCA Space\")\n",
    "plt.colorbar(scatter, label=\"Cluster\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "feed0ce4",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_names = {0: \"Data Junkies\", 1: \"Power Users\",\n",
    "                 2: \"Budget Min.\", 3: \"Trad. Callers\"}\n",
    "plt.figure(figsize=(10, 7))\n",
    "for cid, name in cluster_names.items():\n",
    "    mask = df[\"cluster\"] == cid\n",
    "    plt.scatter(X_pca[mask, 0], X_pca[mask, 1], alpha=0.5, label=name, s=50, edgecolor=\"gray\")\n",
    "    cx, cy = X_pca[mask, 0].mean(), X_pca[mask, 1].mean()\n",
    "    plt.annotate(name, (cx, cy), fontsize=11, fontweight=\"bold\", ha=\"center\", va=\"center\",\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.3\", fc=\"white\", ec=\"gray\", alpha=0.8))\n",
    "plt.xlabel(\"PC1 — Overall Usage / Value →\")\n",
    "plt.ylabel(\"PC2 — Digital ↑ vs. Traditional ↓\")\n",
    "plt.title(\"TeleBari — 4 Customer Segments (PCA Visualization)\")\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7376b15",
   "metadata": {},
   "source": [
    "### 📖 Reading the Plot\n",
    "\n",
    "This single chart shows all 500 subscribers, all 8 features (compressed into 2 PCA axes), and all 4 segments. Power Users are on the far right (highest overall usage). Data Junkies are in the upper area (high digital). Traditional Callers are in the lower area (high voice, low digital). Budget Minimalists are on the left (low everything)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5474566f",
   "metadata": {},
   "source": [
    "### 💬 Discussion 5: Spot the Overlap\n",
    "\n",
    "*Instructor leads — students respond:*\n",
    "\n",
    "- Look at the PCA scatter plot. Which two clusters overlap the most?\n",
    "- Why might they overlap in 2D but be separated in the full 8D space?\n",
    "- If you were Amit and these two clusters overlapped, would you merge them into one segment or keep them separate? Why?\n",
    "\n",
    "**Takeaway**: Clusters that overlap in 2D may be separated on features that PC1 and PC2 don't fully capture — 2 components don't explain 100% of the variance. Keep them separate — the marketing actions are completely different. The 2D plot is for **visualization**, not for decision-making. Always go back to the profile table for business decisions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a274832",
   "metadata": {},
   "source": [
    "### 🎯 Exercise 1: The Full Pipeline\n",
    "\n",
    "- Write the complete clustering + PCA pipeline as a single code block (without running K-selection — assume K=4)\n",
    "- The pipeline should: (1) scale features, (2) run K-Means with K=4, (3) run PCA with 2 components, (4) create the annotated scatter plot\n",
    "- This is the \"recipe\" you'd reuse on any new dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62e83e4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "##### Your code goes here #####\n",
    "\n",
    "# Step 1: Scale\n",
    "\n",
    "# Step 2: K-Means (K=4)\n",
    "\n",
    "# Step 3: PCA (2 components)\n",
    "\n",
    "# Step 4: Scatter plot colored by cluster, labeled axes\n",
    "\n",
    "###### End of Your Code ######"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "732f10e0",
   "metadata": {},
   "source": [
    "### ✅ Solution\n",
    "\n",
    "<details>\n",
    "<summary>Click to reveal</summary>\n",
    "\n",
    "```python\n",
    "# --- Full Clustering + PCA Pipeline ---\n",
    "\n",
    "# Step 1: Scale features\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "\n",
    "# Step 2: K-Means clustering\n",
    "kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)\n",
    "labels = kmeans.fit_predict(X_scaled)\n",
    "\n",
    "# Step 3: PCA for visualization\n",
    "pca = PCA(n_components=2)\n",
    "X_pca = pca.fit_transform(X_scaled)\n",
    "\n",
    "# Step 4: Scatter plot\n",
    "plt.figure(figsize=(10, 7))\n",
    "plt.scatter(X_pca[:, 0], X_pca[:, 1],\n",
    "            c=labels, cmap=\"Set2\", alpha=0.6,\n",
    "            edgecolor=\"gray\", s=50)\n",
    "plt.xlabel(f\"PC1 ({pca.explained_variance_ratio_[0]:.0%} variance)\")\n",
    "plt.ylabel(f\"PC2 ({pca.explained_variance_ratio_[1]:.0%} variance)\")\n",
    "plt.title(\"Customer Segments in PCA Space\")\n",
    "plt.colorbar(label=\"Cluster\")\n",
    "plt.show()\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0c24463",
   "metadata": {},
   "source": [
    "---\n",
    "## 📊 Section 7: Amit's Strategy Report\n",
    "*Turn analysis into action. What does Amit tell the board?*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ed5858e",
   "metadata": {},
   "source": [
    "### 💬 Discussion 6: Amit's Board Presentation\n",
    "\n",
    "*Instructor leads — students respond. Amit has 3 minutes at the board meeting:*\n",
    "\n",
    "1. **How many segments?** Why 4? (Reference both elbow and silhouette)\n",
    "2. **Who are they?** Name each segment and its size. One sentence per segment.\n",
    "3. **The PCA insight**: What are the two axes of customer behavior?\n",
    "4. **The #1 action**: If Amit can only do ONE thing based on this analysis, what should it be?\n",
    "5. **A caveat**: What should Amit warn the board about? (Hint: clusters are hypotheses, not facts)\n",
    "\n",
    "**Takeaway**:\n",
    "1. **4 segments** — both elbow and silhouette peak at K=4, each with a distinct profile.\n",
    "2. Data Junkies (~130), Traditional Callers (~120), Budget Minimalists (~150), Power Users (~100).\n",
    "3. PC1 = overall usage intensity, PC2 = digital vs. traditional communication style.\n",
    "4. Launch a VIP retention program for Power Users — they're the highest-value segment and losing even a few costs more than the other segments combined.\n",
    "5. These segments come from 3 months of data, not customer interviews. K-Means will ALWAYS find K groups — even in random noise. The segments make business sense, but they're a starting hypothesis, not ground truth."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd3b9304",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 🎓 What You Just Did\n",
    "\n",
    "- Loaded 500 subscriber records and explored usage patterns with EDA\n",
    "- Scaled features and ran K-Means clustering (no target variable!)\n",
    "- Chose K=4 using the Elbow method and Silhouette score\n",
    "- Profiled and named 4 customer segments with business meaning\n",
    "- Used PCA to compress 8 features into 2 interpretable dimensions\n",
    "- Visualized all 500 subscribers on one labeled scatter plot\n",
    "- Wrote a board-ready segmentation strategy\n",
    "\n",
    "### The Unsupervised Learning Toolkit\n",
    "\n",
    "```python\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)          # scale first!\n",
    "\n",
    "kmeans = KMeans(n_clusters=4, random_state=42)\n",
    "labels = kmeans.fit_predict(X_scaled)       # no y!\n",
    "\n",
    "pca = PCA(n_components=2)\n",
    "X_pca = pca.fit_transform(X_scaled)         # compress to 2D\n",
    "```\n",
    "\n",
    "Notice: `.fit(X)` only — no y anywhere. That's the unsupervised signature.\n",
    "\n",
    "### 💬 One Last Thing\n",
    "\n",
    "Every model so far had a \"right answer\" — churn labels, car prices, startup valuations. Today, there was no right answer. The algorithm discovered structure on its own, and YOU validated it with business knowledge. That's unsupervised learning — and it's what companies use to understand their customers at scale."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
