PS1

  1. 1

    Load and inspect. Load the Titanic dataset and get a first look: first 5 rows, the shape, .info() and summary statistics, as done with the Sababa Market dataset in Lecture 2.

    Scenario: an intern at a consultancy is given the Titanic dataset by a maritime insurer who wants to know what affected survival.

  2. 2a

    Missing values (2.1). Missing data is one of the most common real-world problems; where and why it is missing affects every downstream decision. Count the missing values in each column.

  3. 2b

    Missing values (2.2, conceptual). Which columns have missing values? For each, suggest a strategy (fill or drop) and explain why in 1 to 2 sentences.

  4. 3a

    Survival overview (3.1). Before looking at features, understand the balance of the target. Is survival a 50/50 split or imbalanced? This matters for interpreting model performance later. Compute the overall survival rate.

  5. 3b

    Survival overview (3.2). Draw a bar chart of survived against not survived.

  6. 4a

    Survival by feature (4.1). Which features seem to influence survival? Show the survival rate by Pclass.

  7. 4b

    Survival by feature (4.2). Show the survival rate by Sex.

  8. 4c

    Survival by feature (4.3, conceptual). Based on the charts, which two features seem most predictive of survival, and what is the real-world explanation?

  9. 5a

    Age distribution (5.1). Age is continuous, so use histograms and boxplots. Draw a histogram of Age.

  10. 5b

    Age distribution (5.2). Draw boxplots of Age by Survived.

  11. 5c

    Age distribution (5.3, conceptual). What difference do you see between survivors and non-survivors, and what might explain it?

  12. 6

    Handle missing values. Deal with the missing values found in Q2. Use the median for Age and the mode for Embarked.

    Order of operations (as stated in the notebook): strictly, you should split into train and test before filling missing values, otherwise fill values such as the median age leak information from the test set into training. The homework fills first and splits afterwards for simplicity.

  13. 7

    Select features and target. Keep only useful columns. Identifiers such as PassengerId and Name carry no pattern; Cabin is mostly empty. Define X and y.

  14. 8

    One-hot encoding. Models need numbers, not text. Convert Sex and Embarked into numeric form with pd.get_dummies() and drop_first=True to avoid redundancy.

  15. 9a

    Train/test split (9.1). Split into training and test sets (test_size=0.2, random_state=42).

  16. 9b

    Train/test split (9.2, conceptual). Why split, and what goes wrong if you evaluate on the training data? 2 to 3 sentences.

  17. 10

    Decision tree. Train a decision tree classifier (hint: DecisionTreeClassifier(random_state=42)).

    All four models in Part C follow the same scikit-learn pattern: create, fit, predict, then score with accuracy_score (fraction of test passengers classified correctly).

  18. 11

    Logistic regression. Train a logistic regression (hint: LogisticRegression(random_state=42, max_iter=1000)). It estimates the probability that a passenger survived and picks the most likely class.

  19. 12

    K-nearest neighbours. KNN predicts by taking a vote among the K closest training passengers. Because it uses distances, large-range features (Fare, 0 to 512) would dominate small-range ones (SibSp, 0 to 8), so scale first with StandardScaler. Use KNeighborsClassifier(n_neighbors=5).

  20. 13

    Random forest. Train a random forest (hint: RandomForestClassifier(random_state=42, n_estimators=100)), the "wisdom of the crowd".

  21. 14a

    Model comparison (14.1). Put all four models side by side in a DataFrame with columns Model and Accuracy.

  22. 14b

    Model comparison (14.2). Draw a bar chart of the accuracies.

  23. 15a

    Confusion matrix and detailed metrics (15.1). Identify the best model and show its confusion matrix.

  24. 15b

    Confusion matrix and detailed metrics (15.2). Print a classification report for the best model.

  25. 15c

    Confusion matrix and detailed metrics (15.3, conceptual). How many passengers did the best model wrongly predict as "survived" when they did not? What is this error called? Why might a maritime insurer care?

  26. 16a

    Feature importance (16.1). Random forests report which features contributed most. Compute and plot feature importances as a horizontal bar chart.

  27. 16b

    Feature importance (16.2, conceptual). Which feature is most important; does that match the EDA? 2 to 3 sentences.

  28. 17a

    Precision against recall (a). The insurer might use the model in two ways. (a) Pricing risk: identify passengers likely to NOT survive, to set higher premiums. Should the company prioritise precision or recall? Explain in 2 to 3 sentences.

  29. 17b

    Precision against recall (b). (b) Safety investment: identify ALL passengers at risk, to invest in safety measures. Should the company prioritise precision or recall? Explain in 2 to 3 sentences.

  30. 18

    Final recommendation. Write 4 to 6 sentences covering: what you found in the data, which model performed best and how well, the key factors that predict survival, and what you would recommend the client focus on.

Related: Lecture 1 - Introduction, Lecture 2 - EDA and Charts, Lecture 3 - Classification, KNN, Random Tree