PS5
- #data-science
- #intro-data-science
- #recitation
- #clustering
- #k-means
- #pca
- #unsupervised-learning
- #customer-segmentation
Toolkit
- 1
The full pipeline.
Write the complete clustering + PCA pipeline as a single code block (do not run K-selection; assume K=4). It should:
- scale features,
- run K-Means with K=4,
- run PCA with 2 components,
- create the scatter plot (coloured by cluster, labelled axes).
This is the "recipe" you would reuse on any new dataset.
Recitation walkthrough
Scale first (mandatory)
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.metrics import silhouette_score
X = df.drop(columns="subscriber_id")
scaler = StandardScaler()
X_scaled = pd.DataFrame(scaler.fit_transform(X), columns=X.columns, index=X.index)
monthly_data_gb has a much wider range than the others; unscaled, it would dominate every distance calculation in K-Means. After scaling every feature has mean about 0 and standard deviation about 1.
K-Means
kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
df["cluster"] = kmeans.fit_predict(X_scaled)
print(kmeans.inertia_) # within-cluster sum of squared distances
K-Means repeats: assign each point to the nearest centroid, move each centroid to its group's mean, until stable. You choose K.
Choosing K: elbow and silhouette
inertias, sil_scores = [], []
for k in range(2, 11):
km = KMeans(n_clusters=k, random_state=42, n_init=10)
labels = km.fit_predict(X_scaled)
inertias.append(km.inertia_)
sil_scores.append(silhouette_score(X_scaled, labels))
The notebook's takeaway: both methods point to K=4: the Elbow Method curve bends at 4 and the Silhouette Score peaks at 4. If the marketing team can only run 3 campaigns, K=3 is mathematically worse but you can merge the two most similar segments: business constraints override the maths, but document the trade-off. K is a business decision aided by maths.
Profiling and naming clusters
profiles = df.groupby("cluster")[X.columns.tolist()].mean().round(1) # on the ORIGINAL scale
profiles["size"] = df["cluster"].value_counts().sort_index()
| Segment | Profile (notebook's takeaway) | Marketing action |
|---|---|---|
| Data Junkies (~130) | High data, high app sessions, low voice and SMS | Unlimited data upgrade, streaming partnerships |
| Traditional Callers (~120) | High voice, international and SMS; low data and app; long tenure | International bundles, family plans |
| Budget Minimalists (~150) | Low everything, lowest bills, short tenure, most support calls | Simple cheap plans, less support friction, loyalty rewards |
| Power Users (~100) | High across the board, highest bills, long tenure, fewest support calls | VIP support, early access, "thank you" perks |
PCA: 8 features to 2 axes
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
print(pca.explained_variance_ratio_)
pca_full = PCA().fit(X_scaled)
cumvar = pca_full.explained_variance_ratio_.cumsum() # how many components do we need?
loadings = pd.DataFrame(pca.components_, columns=X.columns, index=["PC1", "PC2"]).round(2)
- PC1 loads positively on minutes, SMS, bill, account age and international minutes, and negatively on support calls: an "Overall Usage / Value" axis.
- PC2 loads positively on data and app sessions: a "Digital vs Traditional" axis.
- Two components are fine for visualisation; for analysis use the cumulative-variance plot (the notebook draws an 80% line) to choose more. See Explained Variance.
Caveats the notebook stresses
- If two clusters overlap on the 2D PCA plot they may still be separated in the full 8D space, because 2 components do not explain 100% of the variance. Use the 2D plot for visualisation, not decisions; go back to the profile table.
- K-Means will always find K groups, even in random noise. The segments are a starting hypothesis, not ground truth.
Related: Lecture 5 - Clustering and Dimensionality Reduction | Lecture 4 - Regression | Data Science