PS7

  1. 1

    Understand the architecture.

    hidden_layer_sizes=(32,) is one hidden layer with 32 neurons.

    • How many weight connections between the input (8 features) and the hidden layer?
    • How many between the hidden layer (32 neurons) and the output (2 classes)?
    • Does the total match the mlp_simple.coefs_ shapes?
    • Why do more parameters mean more capacity to learn?
  2. 2

    Diagnose overfitting.

    Compare three models: the overfit MLP (256, 128, 64), the early-stopped MLP (64, 32), and the simple MLP (32,). For each print train accuracy and test accuracy. Which has the smallest train-test gap? Which has the best test accuracy? What is the lesson about model size vs dataset size?

  3. 3

    Cross-validation check.

    Run 5-fold cross-validation on the best MLP architecture and on the Decision Tree. Compare mean ± std of accuracy. Is the MLP advantage consistent across folds, or just lucky on one split?

Recitation walkthrough

Scale, split, baselines

python
from sklearn.neural_network import MLPClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier

X = df.drop(columns=["zone_status", "observation_id"])
y = df["zone_status"]

scaler = StandardScaler()           # non-negotiable for MLPs, same reason as KNN
X_scaled = pd.DataFrame(scaler.fit_transform(X), columns=X.columns, index=X.index)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2,
                                                    random_state=42, stratify=y)

baselines = {
    "Logistic Regression": LogisticRegression(max_iter=1000, random_state=42),
    "Decision Tree (d=5)": DecisionTreeClassifier(max_depth=5, random_state=42),
}

The notebook says the best traditional model is the Decision Tree at about 77% accuracy.

The first MLP

python
mlp_simple = MLPClassifier(
    hidden_layer_sizes=(32,),    # one hidden layer of 32 neurons
    activation="relu",           # ReLU in hidden layers
    max_iter=500,                # maximum training epochs
    random_state=42,
)
mlp_simple.fit(X_train, y_train)

Each neuron computes a weighted sum plus an Activation Function; training adjusts the weights to cut prediction error. Useful attributes: coefs_ (weight matrices), intercepts_ (biases), n_iter_, loss_, loss_curve_.

Architecture search and diminishing returns

python
architectures = {"(16,)": (16,), "(32,)": (32,), "(64,)": (64,),
                 "(32, 16)": (32, 16), "(64, 32)": (64, 32), "(64, 32, 16)": (64, 32, 16)}

The notebook's lesson: going from (16,) to (32, 16) helps noticeably, but (64, 32, 16) barely helps while tripling the parameters. (32, 16) is recommended as the best accuracy-per-parameter trade-off, and small enough for phone inference.

Overfitting and early stopping

python
mlp_overfit = MLPClassifier(hidden_layer_sizes=(256, 128, 64), activation="relu",
                            max_iter=2000, random_state=42).fit(X_train, y_train)

mlp_earlystop = MLPClassifier(hidden_layer_sizes=(64, 32), activation="relu", max_iter=2000,
                              early_stopping=True,        # hold out part of training data to watch validation score
                              validation_fraction=0.15,   # 15% of the training set
                              n_iter_no_change=20,        # stop after 20 iterations with no improvement
                              random_state=42).fit(X_train, y_train)

The (256, 128, 64) network has tens of thousands of parameters for only 4,000 training rows (80% of 5,000): it can memorise noise. Its training loss keeps falling while test accuracy does not improve. See Overfitting and Early Stopping.

The decision framework (Lecture 8)

  1. Try simpler models first.
  2. Do you have enough data?
  3. Are there complex feature interactions? (if a linear model works, the MLP adds little)
  4. Can you tolerate a black box? (a Decision Tree is more explainable)
  5. Is inference speed acceptable?

Rotem's recommended fallback if the MLP fails in production: the Decision Tree. The notebook also notes that MLPClassifier is CPU only with fully connected layers; for GPUs, CNNs, RNNs and transformers you move to TensorFlow/Keras or PyTorch.

Related: Lecture 8 - Intro to Deep Learning | Lecture 3 - Classification, KNN, Random Tree | Data Science