Название: Data Engineering Design Patterns: Recipes for Solving the Most Common Data Engineering Problems (Final Release)
Автор: Bartosz Konieczny
Издательство: O’Reilly Media, Inc.
Год: 2025
Страниц: 423
Язык: английский
Формат: epub
Размер: 10.1 MB

Data projects are an intrinsic part of an organization’s technical ecosystem, but data engineers in many companies continue to work on problems that others have already solved. This hands-on guide shows you how to provide valuable data by focusing on various aspects of data engineering, including data ingestion, data quality, idempotency, and more.

Author Bartosz Konieczny guides you through the process of building reliable end-to-end data engineering projects, from data ingestion to data observability, focusing on data engineering design patterns that solve common business problems in a secure and storage-optimized manner. Each pattern includes a user-facing description of the problem, solutions, and consequences that place the pattern into the context of real-life scenarios.

You’re about to replace a legacy data processing framework written in the C# programming language, which nobody in your organization knows anymore. All the maintainers left the company without leaving any useful documentation. You’ve performed a reverse-engineering step, and now, you are rewriting the logic with a modern open source Python library. At this point, you need to migrate the pipelines, but since your reverse-engineering approach may not be perfect, you prefer to keep the old pipelines running until their consumers don’t switch to the new solution. Therefore, during the migration, you’ll need to write the processed dataset in two different places.

Throughout this journey, you’ll use open source data tools and public cloud services to apply each pattern. You'll learn:

Challenges data engineers face and their impact on data systems
How these challenges relate to data system components
Useful applications of data engineering patterns
How to identify and fix issues with your current data components
TTechnology-agnostic solutions to new and existing data projects, with open source implementation examples

Bartosz Konieczny is a freelance data engineer who's been coding since 2010. He's held various senior hands-on positions that allowed him to work on many data engineering problems in batch and stream processing.