Название: Machine Learning Systems at Scale, Vol 2
Автор: Vijay Janapa Reddi
Издательство: The MIT Press
Год: 2027
Страниц: 1170
Язык: английский
Формат: pdf (true)
Размер: 17.9 MB

Modern Machine Learning operates at scales that fundamentally change engineering requirements—models too large for single GPUs, services spanning continents, deployments carrying societal responsibilities. This book addresses AI engineering at scale. The treatment follows the lifecycle of a massive-scale system: defining the distributed architecture, building the physical infrastructure fleet, ensuring operational reliability, deploying to global users, and hardening the system for safety and responsibility.

For workloads already written in PyTorch, torch.compile is the least disruptive compiler intervention: it tries to capture enough static graph to fuse memory-bound regions while preserving PyTorch’s dynamic Python execution model. It operates through three components: TorchDynamo for graph capture, TorchInductor for code generation, and AOTAutograd for ahead-of-time backward graph construction when training workloads need compiled backward passes. TorchDynamo operates at the Python bytecode level9 , a design choice that distinguishes it from earlier tracing approaches. Previous tracing methods (torch.jit.trace, torch.fx) operated at the Python source or abstract syntax tree (AST) level, requiring users to avoid unsupported Python constructs. TorchDynamo intercepts the bytecode interpreter itself, capturing a computational graph without requiring the user to modify their model code. When TorchDynamo encounters Python constructs it cannot trace (data-dependent control flow, unsupported operations), it inserts a graph break that splits the trace into multiple subgraphs, each compiled independently. The goal is to capture as large a subgraph as possible while gracefully handling dynamic Python behavior.

What You Will Learn:

The four parts extend ML systems foundations into production-scale systems by following the Fleet Stack from bottom to top:

Part I: The Fleet—Build the physical computer. Architect the datacenter infrastructure, high-bandwidth network fabrics, and scalable data storage that form the foundation of every distributed ML deployment.
Part II: Distributed ML—Master the algorithms of scale. Learn how to coordinate computation across thousands of devices using parallelism strategies, collective communication primitives, fault tolerance mechanisms, and fleet orchestration.
Part III: Deployment at Scale—Serve the world. Navigate the shift from training to inference, optimize performance across the serving stack, push intelligence to the edge, and manage the operational lifecycle of production fleets.
Part IV: The Responsible Fleet—Harden and govern the system. Address security, robustness, environmental sustainability, and responsible engineering in large-scale operations.

Prerequisites:

This book assumes:

Foundational or equivalent background in single-machine ML systems
Fluency in Python, including functions, classes, and data manipulation with NumPy. Familiarity with at least one ML framework (PyTorch, TensorFlow, or JAX)
Mathematics foundations in linear algebra, calculus, and probability
Familiarity with distributed systems concepts (networking, parallelism) is helpful for advanced topics

Who This Book Is For:
This book is for anyone who understands Machine Learning on a single machine and now faces the challenge of making it work at scale. The problems of scale are not the problems of a single node, only bigger. They are qualitatively different: networks partition, hardware fails as a statistical certainty, and the societal impact of every design decision is amplified by millions of users.

The scope ranges from designing the physical substrate of an AI data center, to scaling training beyond the limits of a single accelerator, to reasoning about what happens when a production fleet serves a global user base. Throughout, the focus is on the physics of distribution—the invariants that govern every fleet-scale ML system regardless of the framework, the model, or the era.