The ML System Design Interview

Intermediate · 24 min read · High-Level Design, ML & AI Systems

Turn a product goal into a learning problem: framing, metrics, labels, features, models, serving and monitoring.

An ML system design round asks you to design a product feature whose core is a learned model: "design the Instagram feed", "design YouTube recommendations", "detect harmful content", "predict which ad a user will click".

Many candidates fail this round in one of two ways. Strong backend engineers draw a beautiful serving architecture around a box labelled "model" and never say what it predicts.

How this round differs from system design

The last two rows explain why this round exists. A bug in a web service throws exceptions; a bug in an ML system returns plausible-looking results that are slightly worse every day, and nobody notices for months.

The framework

Spend roughly this much of a 45-minute round on each step. The order matters, because each step constrains the next.

The mistakes interviewers see most

Levels

Interview questions

  • Why is click-through rate often a poor objective for a feed?
  • Your new model improves offline AUC by 2% but the A/B test shows no change in engagement. What could explain it?
  • What is training–serving skew and how do you prevent it?
  • How do you handle a brand-new user with no history?

Practice

Write your answer before opening the suggested one. Anything you mark "Partly" or "Missed it" comes back in your review deck.

Further reading

Key takeaways

  • Frame the problem before choosing a model: input, output, task type, and how the prediction is used.
  • Choose proxies deliberately; clicks alone reward clickbait. Combine several predicted outcomes in a tunable value model.
  • Use offline metrics to filter models and online A/B tests to decide launches, with guardrails.
  • Name the label problems: position bias, selection bias, label delay, leakage and imbalance.
  • Large systems are funnels: cheap candidate generation, a heavier ranker, then re-ranking for diversity and rules.
  • A feature store and logged serving features are the main defence against training–serving skew.
  • ML systems degrade silently, so monitor data, drift and sliced online metrics, and retrain on a schedule.

Previous: Design a Distributed File System (GFS/HDFS) · Next: Design a Recommendation & Feed Ranking System