PS1
- #data-science
- #intro-data-science
- #problem-set
- #classification
- #eda
- #preprocessing
- #titanic
- #scikit-learn
Toolkit
- 1
Load and inspect. Load the Titanic dataset and get a first look: first 5 rows, the shape,
.info()and summary statistics, as done with the Sababa Market dataset in Lecture 2.Scenario: an intern at a consultancy is given the Titanic dataset by a maritime insurer who wants to know what affected survival.
- 2a
Missing values (2.1). Missing data is one of the most common real-world problems; where and why it is missing affects every downstream decision. Count the missing values in each column.
- 2b
Missing values (2.2, conceptual). Which columns have missing values? For each, suggest a strategy (fill or drop) and explain why in 1 to 2 sentences.
- 3a
Survival overview (3.1). Before looking at features, understand the balance of the target. Is survival a 50/50 split or imbalanced? This matters for interpreting model performance later. Compute the overall survival rate.
- 3b
Survival overview (3.2). Draw a bar chart of survived against not survived.
- 4a
Survival by feature (4.1). Which features seem to influence survival? Show the survival rate by
Pclass. - 4b
Survival by feature (4.2). Show the survival rate by
Sex. - 4c
Survival by feature (4.3, conceptual). Based on the charts, which two features seem most predictive of survival, and what is the real-world explanation?
- 5a
Age distribution (5.1). Age is continuous, so use histograms and boxplots. Draw a histogram of
Age. - 5b
Age distribution (5.2). Draw boxplots of
AgebySurvived. - 5c
Age distribution (5.3, conceptual). What difference do you see between survivors and non-survivors, and what might explain it?
- 6
Handle missing values. Deal with the missing values found in Q2. Use the median for
Ageand the mode forEmbarked.Order of operations (as stated in the notebook): strictly, you should split into train and test before filling missing values, otherwise fill values such as the median age leak information from the test set into training. The homework fills first and splits afterwards for simplicity.
- 7
Select features and target. Keep only useful columns. Identifiers such as
PassengerIdandNamecarry no pattern;Cabinis mostly empty. DefineXandy. - 8
One-hot encoding. Models need numbers, not text. Convert
SexandEmbarkedinto numeric form withpd.get_dummies()anddrop_first=Trueto avoid redundancy. - 9a
Train/test split (9.1). Split into training and test sets (
test_size=0.2,random_state=42). - 9b
Train/test split (9.2, conceptual). Why split, and what goes wrong if you evaluate on the training data? 2 to 3 sentences.
- 10
Decision tree. Train a decision tree classifier (hint:
DecisionTreeClassifier(random_state=42)).All four models in Part C follow the same scikit-learn pattern: create, fit, predict, then score with
accuracy_score(fraction of test passengers classified correctly). - 11
Logistic regression. Train a logistic regression (hint:
LogisticRegression(random_state=42, max_iter=1000)). It estimates the probability that a passenger survived and picks the most likely class. - 12
K-nearest neighbours. KNN predicts by taking a vote among the K closest training passengers. Because it uses distances, large-range features (
Fare, 0 to 512) would dominate small-range ones (SibSp, 0 to 8), so scale first withStandardScaler. UseKNeighborsClassifier(n_neighbors=5). - 13
Random forest. Train a random forest (hint:
RandomForestClassifier(random_state=42, n_estimators=100)), the "wisdom of the crowd". - 14a
Model comparison (14.1). Put all four models side by side in a DataFrame with columns
ModelandAccuracy. - 14b
Model comparison (14.2). Draw a bar chart of the accuracies.
- 15a
Confusion matrix and detailed metrics (15.1). Identify the best model and show its confusion matrix.
- 15b
Confusion matrix and detailed metrics (15.2). Print a classification report for the best model.
- 15c
Confusion matrix and detailed metrics (15.3, conceptual). How many passengers did the best model wrongly predict as "survived" when they did not? What is this error called? Why might a maritime insurer care?
- 16a
Feature importance (16.1). Random forests report which features contributed most. Compute and plot feature importances as a horizontal bar chart.
- 16b
Feature importance (16.2, conceptual). Which feature is most important; does that match the EDA? 2 to 3 sentences.
- 17a
Precision against recall (a). The insurer might use the model in two ways. (a) Pricing risk: identify passengers likely to NOT survive, to set higher premiums. Should the company prioritise precision or recall? Explain in 2 to 3 sentences.
- 17b
Precision against recall (b). (b) Safety investment: identify ALL passengers at risk, to invest in safety measures. Should the company prioritise precision or recall? Explain in 2 to 3 sentences.
- 18
Final recommendation. Write 4 to 6 sentences covering: what you found in the data, which model performed best and how well, the key factors that predict survival, and what you would recommend the client focus on.
Related: Lecture 1 - Introduction, Lecture 2 - EDA and Charts, Lecture 3 - Classification, KNN, Random Tree