IDC Revision

Sample Exam — 25 Practice Questions · worked-solution

Sample Exam — 25 Practice Questions (Loan Pipeline)

Original paper ↗
What this paper is

The lecturer's sample exam — 25 multiple-choice questions, every one of them about a single deliberately-flawed notebook, loan_pipeline.ipynb. The exam is expected to follow this format exactly: the code is released in advance, and you are expected to know it line by line.

No official answer key was supplied

The sample paper ends by telling you to "check your answers against the version of this file that includes the answer key" — that version was not distributed. Every answer below is derived, grounded line-by-line in the notebook and the brief rather than copied from a key. The reasoning is shown in full precisely so you can check it rather than trust it, which is the habit the exam is built to reward.

How the paper is structured

Part Questions What it tests The move it wants
1 — Is the concern valid? 1–10 How closely you read the code Judge a claim against the source, including claims that are true but trivial and claims that are right for the wrong reason
2 — Approve the right plan 11–20 Whether plausible-sounding plans can mislead you Pick the fix that addresses the cause, not the symptom — and accept the cost it carries
3 — The model decision 21–25 Whether you set criteria before reading numbers Commit to criteria first, then apply them even when the headline metric ties

Count your mistakes by part, not in total — each part fails in a different way, and knowing which part you lose marks in tells you what to revise.

The recurring answer shape

Across all three parts the correct option is almost always the one that is specific, calibrated, and willing to name a cost:

  • It points at a named column, line, or number rather than a general principle.
  • It is hedged where reality is hedged — "counts materially in favour of", "not valid as a priority", "for now". Absolute options ("always", "entirely", "rules out", "no bearing") are almost always wrong.
  • It accepts a downside: a lower score, a smaller dataset, real money spent, loans excluded. Wrong options tend to promise a fix with no cost attached.

The distractors are correspondingly patterned. Expect: a true statement attached to a wrong conclusion (Q7, Q10), a fix that treats the symptom (Q17 B), a fix that sounds sophisticated but makes things worse (Q11 D, Q20 D), and one option that quietly carries the leak through the fix (Q16 B).

Before you attempt it

Read the Loan Pipeline — Code Walkthrough & Defect Catalogue first if you have not already. Fourteen of these twenty-five questions turn on something you can only see by reading load_data closely — which columns are generated from the label, how many rows each customer gets, and where the missing values come from.

  1. Q1 — Columns that are empty on decision day

    A member of your team raises the following concern about the pipeline:

    "Two of the columns will simply be empty when a new customer applies for a loan."

    Reviewing the code and its outputs yourself — is this concern valid?
  2. Q2 — One customer, several rows

    A member of your team raises the following concern about the pipeline:

    "A single customer can occupy several rows of the table, which the split does not take into account."

    Reviewing the code and its outputs yourself — is this concern valid?
  3. Q3 — Loans observed for different lengths of time

    A member of your team raises the following concern about the pipeline:

    "Loans have been observed for different lengths of time, yet they are all labelled by the same rule."

    Reviewing the code and its outputs yourself — is this concern valid?
  4. Q4 — No rejected applicants in the data

    A member of your team raises the following concern about the pipeline:

    "There are no examples in the data of the applicants the bank turned down."

    Reviewing the code and its outputs yourself — is this concern valid?
  5. Q5 — Income holding two kinds of number

    A member of your team raises the following concern about the pipeline:

    "The income column looks as though it holds two different kinds of number."

    Reviewing the code and its outputs yourself — is this concern valid?
  6. Q6 — Does the pipeline use a validation set?

    A member of your team raises the following concern about the pipeline:

    "The pipeline uses a validation set to decide when to stop training the neural network."

    Reviewing the code and its outputs yourself — is this concern valid?
  7. Q7 — Standardising features for a Random Forest

    A member of your team raises the following concern about the pipeline:

    "The Random Forest's results are invalid because its features were standardised before training."

    Reviewing the code and its outputs yourself — is this concern valid?
  8. Q8 — 'Model parameters' in the comments

    A member of your team raises the following concern about the pipeline:

    "The notebook's comments describe the number of trees and the number of epochs as things the model works out for itself."

    Reviewing the code and its outputs yourself — is this concern valid?
  9. Q9 — Do the two models agree?

    A member of your team raises the following concern about the pipeline:

    "The pipeline never checks whether the two models actually agree with each other."

    Reviewing the code and its outputs yourself — is this concern valid?
  10. Q10 — 'It was written by an AI assistant'

    A member of your team raises the following concern about the pipeline:

    "The results cannot be trusted because the code was written with the help of an AI assistant."

    Reviewing the code and its outputs yourself — is this concern valid?
  11. Q11 — Plan: customers on both sides of the split

    The following concern is real and confirmed. The data team proposes four ways forward:

    "The same customers appear on both sides of the split, so part of the reported score reflects recognition rather than prediction."

    Which plan do you approve?
  12. Q12 — Plan: features that describe events after the loan

    The following concern is real and confirmed. The data team proposes four ways forward:

    "Two features describe events that occur after the loan is granted, and will be empty for every new applicant."

    Which plan do you approve?
  13. Q13 — Plan: the imputation average was computed too early

    The following concern is real and confirmed. The data team proposes four ways forward:

    "The average used to fill missing incomes was computed over the whole table, before the data was split."

    Which plan do you approve?
  14. Q14 — Plan: credit scores filled with zero

    The following concern is real and confirmed. The data team proposes four ways forward:

    "Missing credit scores were replaced with 0 — a value that cannot occur on a 300–850 scale."

    Which plan do you approve?
  15. Q15 — Plan: a label with no observation window

    The following concern is real and confirmed. The data team proposes four ways forward:

    "The label 'missed 3+ payments at some point' has no fixed observation window."

    Which plan do you approve?
  16. Q16 — Plan: rows per customer-month vs decisions per application

    The following concern is real and confirmed. The data team proposes four ways forward:

    "The table holds one row per customer-month, while the bank makes one decision per application."

    Which plan do you approve?
  17. Q17 — Plan: one column, two units

    The following concern is real and confirmed. The data team proposes four ways forward:

    "One column carries two different units, depending on which source system produced the record."

    Which plan do you approve?
  18. Q18 — Plan: more trees, more epochs

    The following concern is real and confirmed. The data team proposes four ways forward:

    "The team asks to raise the number of trees and the number of training epochs in order to improve the results."

    Which plan do you approve?
  19. Q19 — Plan: one accuracy figure on imbalanced data

    The following concern is real and confirmed. The data team proposes four ways forward:

    "The pipeline reports one accuracy figure per model, on data where about 14% of customers default."

    Which plan do you approve?
  20. Q20 — Plan: a model for the whole applicant population

    The following concern is real and confirmed. The data team proposes four ways forward:

    "The bank wants a model that works for the whole applicant population, not only for profiles the old rules approved."

    Which plan do you approve?
  21. Q21 — Stating the criteria before the numbers

    The model decision:

    "Before any numbers are presented, you must say what would make one model preferable to the other."

    Which statement of criteria is defensible?
  22. Q22 — A recommendation that contradicts its own printout

    The model decision:

    "The pipeline's closing line recommends the neural network 'because deep learning is the more advanced technology' — although the Random Forest scored higher in the team's own printout."

    What is the significance of this?
  23. Q23 — The regulator's explainability requirement

    The model decision:

    "The regulator requires the bank to explain, in plain terms, why any particular application was rejected."

    How does this requirement bear on the choice between the two models?
  24. Q24 — Asymmetric costs and the threshold

    The model decision:

    "A missed defaulter costs the bank roughly the outstanding principal; a wrongly rejected applicant costs the interest margin that customer would have generated."

    What follows from this asymmetry?
  25. Q25 — The corrected evaluation arrives

    The model decision:

    "The corrected evaluation arrives: Random Forest — 36% of defaulters caught, easy to explain, cheap to run. Neural Network — 37% caught, requires additional explanation tooling and more infrastructure."

    What is the defensible call?