Lecture 3 — Classification

Part of: Data Science Previous: Lecture 2 - EDA and Charts | Next: Lecture 4 - Regression Key concepts: Supervised Learning, Decision Tree, Logistic Regression, KNN, Random Forest, Confusion Matrix, Precision and Recall

The Core Idea

Classification = predicting a discrete label from features. Regression predicts a number; classification predicts a category.

Running case study

Yalla Fitness — 300 members, 30% churn. Goal: predict which members will cancel next month from age, visit frequency, tenure, membership type, etc. Each row = one member; label = churn ∈ {0, 1}.

Binary vs. multiclass

The vocabulary

• Features (X): the inputs — age, visits, tenure, price paid…
• Label (y): the thing being predicted — churn.
• Decision boundary: the surface in feature space that separates classes.
• Training set: data the model learns from.
• Test set: held-out data used to measure honest performance.

The Supervised-Learning Pipeline

from sklearn.model_selection import train_test_split

X = df.drop(columns=["churn"])
y = df["churn"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Never test on training data

A model that memorises the training set can score 100% there but fail on new members. The test set exists to catch this.

stratify=y keeps the churn rate the same in both splits — crucial when classes are imbalanced.

Algorithm 1 — Decision Tree

Intuition

A flowchart of if/else splits. At each node, pick the feature + threshold that best separates the classes.

              visits_per_month < 4 ?
                /              \
              yes              no
              /                  \
         tenure < 3m ?         churn = 0
          /        \
        yes        no
        /            \
     churn=1       churn=0

How a split is chosen — Gini impurity

where $p_k$ is the proportion of class $k$ in node $S$. Gini = 0 for a pure node, 0.5 at maximum impurity (binary). The tree picks the split that minimises weighted Gini of the children.

Code

from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(max_depth=4, random_state=42)
clf.fit(X_train, y_train)
preds = clf.predict(X_test)

Strengths & weaknesses

Overfitting

An unrestricted tree grows until every leaf is pure — it memorises noise. Use max_depth, min_samples_leaf, or ensemble it (→ Random Forest).

Handling categoricals — one-hot encoding

X = pd.get_dummies(df, columns=["membership_type"])

Turns membership_type ∈ {basic, premium, student} into three 0/1 columns.

Algorithm 2 — Logistic Regression

The sigmoid

Linear regression outputs any real number; we need a probability in [0, 1]. Pass a linear score through the sigmoid:

$P(\text{churn} = 1 \mid X) = \sigma(z)$. Classify as 1 if $P > 0.5$ (threshold tunable).

Code

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

clf = LogisticRegression(max_iter=1000)
clf.fit(X_train_s, y_train)
probs = clf.predict_proba(X_test_s)[:, 1]

Strengths & weaknesses

Why probabilities matter

"80% chance to churn" lets the business prioritise who to call. A hard 0/1 prediction loses this nuance.

Algorithm 3 — K-Nearest Neighbours (KNN)

Intuition

"Tell me who your neighbours are and I'll tell you who you are." To classify a new member, find the $K$ closest members in feature space and take a majority vote.

Distance (Euclidean):

Code

from sklearn.neighbors import KNeighborsClassifier
clf = KNeighborsClassifier(n_neighbors=5)
clf.fit(X_train_s, y_train)   # use scaled features!

Choosing K

Always scale first

KNN uses raw distance — a feature measured in thousands of shekels will dominate a feature measured in visits per week. StandardScaler or MinMaxScaler before KNN. (Same is true for logistic regression with regularisation.)

Strengths & weaknesses

Try it: step through the graph

type: knn-classifier

Algorithm 4 — Random Forest

Intuition

One tree is unstable. A forest of many different trees, each trained on a random sample of data + features, then voting is far more robust. This is an ensemble.

Recipe

1. Take $B$ bootstrap samples of the training set.
2. On each, grow a tree using a random subset of features at each split.
3. For prediction, let every tree vote — majority wins (or average probabilities).

Code

from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_estimators=200, max_depth=8, random_state=42)
clf.fit(X_train, y_train)
importances = pd.Series(clf.feature_importances_, index=X.columns).sort_values(ascending=False)

Strengths & weaknesses

Why ensembles win

Each tree makes different mistakes. Averaging cancels the noise while preserving the signal — the bias stays roughly the same but variance drops.

Evaluating a Classifier

Accuracy — and why it lies

The imbalance trap

If only 5% of members churn, predicting "no one churns" gets 95% accuracy — and is completely useless. Accuracy is safe only when classes are roughly balanced.

Confusion matrix

from sklearn.metrics import confusion_matrix, classification_report
print(confusion_matrix(y_test, preds))
print(classification_report(y_test, preds))

Precision, recall, F1

Precision–recall trade-off

Lower the threshold → more positives → recall ↑, precision ↓. Raise it → the opposite. Pick the threshold that matches the business cost of each error.

Model Comparison (Yalla Fitness results)

Random Forest wins overall — but if calling members is cheap, you might pick the model with highest recall even at lower precision.

The scikit-learn API — One Shape

Every classifier in sklearn uses the same three calls:

model = SomeClassifier(...)      # 1. create
model.fit(X_train, y_train)      # 2. train
preds  = model.predict(X_test)   # 3. predict
probs  = model.predict_proba(X_test)   # (if supported)

Swapping models = changing one line. Build the pipeline first; iterate on the model later.

Summary — The Classification Workflow

1. Define the label — binary or multiclass; check class balance.
2. Prep features — one-hot categoricals, scale numerics for KNN / logistic regression.
3. Split — train_test_split(..., stratify=y).
4. Train several models — Tree, Logistic, KNN, Random Forest.
5. Evaluate — confusion matrix, precision, recall, F1 (not just accuracy).
6. Tune threshold to match the business cost of FP vs FN.
7. Pick the winner — usually Random Forest, but match to the decision problem.

Related Notes

• Previous: Lecture 2 - EDA and Charts
• Next: Lecture 4 - Regression
• Foundational: Supervised Learning, Train Test Split, One-Hot Encoding, Feature Scaling
• Related: Gini Impurity, Sigmoid Function, Ensemble Methods, Bias-Variance Tradeoff