Designing Machine Learning Systems

What it’s about

Designing Machine Learning Systems by Chip Huyen offers the comprehensive, end-to-end guide that ML engineers and data scientists have long needed to bridge the gap between academic model-building and the messy realities of production. Rather than treating the ML algorithm as the centerpiece, Huyen situates it as just one small component within a much larger system encompassing business objectives, data pipelines, feature engineering, deployment infrastructure, monitoring, and responsible AI. Drawing on her experience at NVIDIA, Netflix, Snorkel AI, and Stanford—where she teaches the course CS 329S: Machine Learning Systems Design—she walks readers through every stage of the ML project lifecycle with concrete case studies, trade-off discussions, and practical frameworks. The book covers everything from sampling strategies and labeling techniques, through model development and offline evaluation, to online prediction, data distribution shift detection, continual learning, MLOps infrastructure, and the human and ethical dimensions of deploying AI at scale. Whether you are deploying your first model or managing hundreds in production, this book provides the principled vocabulary and decision-making framework to do it right.

The through-line

Who it’s for

ML engineers, data scientists, and engineering managers who want to build ML systems that actually work reliably in production—not just in notebooks—and who are frustrated by the gap between academic ML and real-world deployment complexity.

The problem

Their models perform well in development but degrade, fail silently, or cause harmful outcomes once deployed to real users at scale. They feel overwhelmed, under-equipped, and uncertain about what to do next when production systems break in ways that unit tests and accuracy scores never revealed.

The plan

1. Establish business objectives and translate them into ML objectives and system requirements (reliability, scalability, maintainability, adaptability).
2. Frame the ML problem correctly: choose the right task type, objective function, and decoupled multi-objective structure.
3. Master data engineering fundamentals: data sources, formats, storage engines, and dataflow modes.
4. Create high-quality training data through principled sampling, labeling strategies (natural labels, weak supervision, active learning), and handling class imbalance and data augmentation.
5. Engineer features carefully, avoiding data leakage, prioritizing feature importance and generalization.

The payoff

ML models remain accurate and reliable long after deployment, with degradation detected and corrected quickly. · Teams move from manual, months-long model update cycles to automated, data-driven retraining triggered by real performance signals. · Data scientists own the full ML lifecycle confidently, supported by infrastructure that abstracts away operational complexity.

See our guide

• Build AI Applications →
• Build AI Systems That Ship →
• The Four-S Discipline — One System, Four Capabilities →
• AI Safety, Governance & Alignment Practice (FIELD GUIDE (ALL LEVELS)) →
• Measure And Run Technology Operations →
• Build The Data And AI Systems Behind Analytics →

Related profiles we’ve built

• The AI Marketing Canvas: A Five-Step AI Plan for Marketers →
• Artificial Intelligence →
• Artificial Intelligence - A Very Short Introduction →
• Effective Data Science Infrastructure →
• Human + Machine →
• Time Series Forecasting Using Foundation Models →
• Understanding Deep Learning →
• The basic principle of people analytics learn how to use hr data to drive better outcomes for your business and employees →

Additional reading

• Designing Data-Intensive Applications · Martin Kleppmann

  Recommended for a deeper understanding of data engineering from a systems perspective, covering fundamentals of data storage, processing, and distributed systems.

• Weapons of Math Destruction · Cathy O'Neil

  Recommended for its powerful examples of how poorly designed and biased algorithms can cause harm at scale in the real world.

• Reliable Machine Learning · Cathy O'Neil, Todd Underwood, et al.

  Recommended for learning how to apply traditional software engineering principles of reliability (SRE practices) to ML systems.

• Trustworthy Online Controlled Experiments (A Practical Guide to A/B Testing) · Ron Kohavi, et al.

  Recommended for a thorough guide to the theory and practice of A/B testing and other online experimentation methods.

• Effective Data Science Infrastructure · Ville Tuulos

  Cited for its argument on the importance of a well-designed development environment for improving data scientist productivity.

Featured in these guides

• Build AI Applications →Pole in “Model Quality Versus Organizational Adoption”
• Build AI Systems That Ship →Pole in “Feedback Loops: Flywheel or Trap”
• AI Safety, Governance & Alignment Practice (FIELD GUIDE (ALL LEVELS)) →Cited source
• Measure And Run Technology Operations →Cited source
• Build The Data And AI Systems Behind Analytics →Cited source

Where other books disagree

• Feedback Loops: Flywheel or Trap →
• Model Quality Versus Organizational Adoption →