PS7
- #data-science
- #intro-data-science
- #recitation
- #neural-networks
- #mlp
- #overfitting
- #early-stopping
- #cross-validation
Toolkit
- 1
Understand the architecture.
hidden_layer_sizes=(32,)is one hidden layer with 32 neurons.- How many weight connections between the input (8 features) and the hidden layer?
- How many between the hidden layer (32 neurons) and the output (2 classes)?
- Does the total match the
mlp_simple.coefs_shapes? - Why do more parameters mean more capacity to learn?
- 2
Diagnose overfitting.
Compare three models: the overfit MLP (256, 128, 64), the early-stopped MLP (64, 32), and the simple MLP (32,). For each print train accuracy and test accuracy. Which has the smallest train-test gap? Which has the best test accuracy? What is the lesson about model size vs dataset size?
- 3
Cross-validation check.
Run 5-fold cross-validation on the best MLP architecture and on the Decision Tree. Compare mean ± std of accuracy. Is the MLP advantage consistent across folds, or just lucky on one split?
Recitation walkthrough
Scale, split, baselines
from sklearn.neural_network import MLPClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
X = df.drop(columns=["zone_status", "observation_id"])
y = df["zone_status"]
scaler = StandardScaler() # non-negotiable for MLPs, same reason as KNN
X_scaled = pd.DataFrame(scaler.fit_transform(X), columns=X.columns, index=X.index)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2,
random_state=42, stratify=y)
baselines = {
"Logistic Regression": LogisticRegression(max_iter=1000, random_state=42),
"Decision Tree (d=5)": DecisionTreeClassifier(max_depth=5, random_state=42),
}
The notebook says the best traditional model is the Decision Tree at about 77% accuracy.
The first MLP
mlp_simple = MLPClassifier(
hidden_layer_sizes=(32,), # one hidden layer of 32 neurons
activation="relu", # ReLU in hidden layers
max_iter=500, # maximum training epochs
random_state=42,
)
mlp_simple.fit(X_train, y_train)
Each neuron computes a weighted sum plus an Activation Function; training adjusts the weights to cut prediction error. Useful attributes: coefs_ (weight matrices), intercepts_ (biases), n_iter_, loss_, loss_curve_.
Architecture search and diminishing returns
architectures = {"(16,)": (16,), "(32,)": (32,), "(64,)": (64,),
"(32, 16)": (32, 16), "(64, 32)": (64, 32), "(64, 32, 16)": (64, 32, 16)}
The notebook's lesson: going from (16,) to (32, 16) helps noticeably, but (64, 32, 16) barely helps while tripling the parameters. (32, 16) is recommended as the best accuracy-per-parameter trade-off, and small enough for phone inference.
Overfitting and early stopping
mlp_overfit = MLPClassifier(hidden_layer_sizes=(256, 128, 64), activation="relu",
max_iter=2000, random_state=42).fit(X_train, y_train)
mlp_earlystop = MLPClassifier(hidden_layer_sizes=(64, 32), activation="relu", max_iter=2000,
early_stopping=True, # hold out part of training data to watch validation score
validation_fraction=0.15, # 15% of the training set
n_iter_no_change=20, # stop after 20 iterations with no improvement
random_state=42).fit(X_train, y_train)
The (256, 128, 64) network has tens of thousands of parameters for only 4,000 training rows (80% of 5,000): it can memorise noise. Its training loss keeps falling while test accuracy does not improve. See Overfitting and Early Stopping.
The decision framework (Lecture 8)
- Try simpler models first.
- Do you have enough data?
- Are there complex feature interactions? (if a linear model works, the MLP adds little)
- Can you tolerate a black box? (a Decision Tree is more explainable)
- Is inference speed acceptable?
Rotem's recommended fallback if the MLP fails in production: the Decision Tree. The notebook also notes that MLPClassifier is CPU only with fully connected layers; for GPUs, CNNs, RNNs and transformers you move to TensorFlow/Keras or PyTorch.
Related: Lecture 8 - Intro to Deep Learning | Lecture 3 - Classification, KNN, Random Tree | Data Science