Preparation Questions — Loan Pipeline (15 Questions)

Overfitting and underfitting.

What is overfitting, and what is underfitting? For each one, describe what you would see when you compare performance on the training data with performance on the test data, and give one way to fix it.

Bias.

What does it mean to say a model has high bias? How does this relate to how complex the model is, and how does it relate to underfitting?

Accuracy and baselines.

About 14% of the loans in our data end in default. What accuracy would you get by simply predicting "no default" for every customer? Given that, why is accuracy a weak way to judge this model, and which measures would show you how it actually handles defaulters?

Regression vs classification.

What is the difference between regression and classification? Give one banking example of each, and say which of the two the loan-approval problem is.

Deep learning.

What is deep learning, and when is a neural network the better choice? For a table of about 6,500 rows and 7 columns, would you expect a neural network or a tree-based model to do better, and why?

Ensembles.

What is an ensemble of models? Explain simply why combining many decision trees usually works better than a single tree, and what makes the individual trees in a Random Forest different from one another.

Parameter vs hyperparameter.

What is the difference between a parameter and a hyperparameter? Give two examples of each from loan_pipeline.ipynb, and say where hyperparameters should be chosen.

Train, validation and test.

What is each of the three sets — training, validation and test — for? What goes wrong if you choose your settings by testing them on the test set, and what goes wrong if there is no validation set at all, as in our pipeline?

Data leakage.

What is data leakage? Give the general definition, then name the columns in loans.csv that leak and explain exactly why each one cannot be used to decide on a new applicant.

What a row represents.

What does one row of loans.csv represent, and how many rows does a single customer have? Explain why splitting the data randomly by row is a problem here, and what kind of split solves it.

Missing values.

In the pipeline, missing income is filled with the mean of the whole table and missing credit_score is filled with 0. Give three separate problems with those two lines, and say what should be done instead.

Order of operations.

Why is it a mistake to fit a StandardScaler, or to compute an imputation value, before splitting the data? What is the correct order, and what is this kind of mistake called?

Reading the data.

The income histogram produced by the pipeline shows two separate peaks, one about twelve times the other. What is the most likely cause, and what should be done about it before any modelling continues?

How the data was collected.

Our training data contains only loans that the bank approved in the past. What is this problem called, why does it matter when the model goes into production, and can it be fixed by changing the code?

Choosing between two models.

The pipeline reports about 99% for the Random Forest and 98% for the neural network, then recommends the neural network because "deep learning is the more advanced technology". List the criteria you would actually use to choose between two models, and explain why the comparison chart — whose y-axis runs from 0.90 to 1.00 — is misleading.