Название: Data Engineering with GCP: Practical guide to designing and deploying scalable data pipelines on Google Cloud
Автор: Mahesh T V
Издательство: BPB Publications
Год: 2026
Страниц: 338
Язык: английский
Формат: epub (true)
Размер: 14.0 MB

Google Cloud Platform (GCP) has emerged as a premier leader in cloud analytics, making data engineering skills more critical than ever for modern business success. The current evolution of Generative Artificial Intelligence (AI) and Agentic AI has created a significant demand in the data engineering discipline since the accuracy and effectiveness of AI output primarily depend on the quality of data. Ensuring high-quality, curated data requires a robust and scalable data engineering platform that can cater to the velocity, veracity, and volume of data.

Google is a pioneer in data engineering solutions, which are provided through its GCP. Many of the Fortune 500 companies leverage GCP’s services for transforming petabytes of data for analytics, AI, and Machine Learning (ML). This book begins with data engineering essentials like ETL, ELT, and Big Data roles before moving into GCP environment setup and security. You will learn BigQuery for data warehousing and SQL optimization, followed by real-time ingestion using Pub/Sub, Dataflow, and Datastream. You will learn to integrate Machine Learning via Vertex AI pipelines. Finally, it will provide the skills to use the processed data for analytics, AI, and ML use cases.

The book is structured to start with the key foundation concepts of data engineering. It covers a wide range of GCP data engineering services for data ingestion, data storage, data warehouse and data transformation. The book also explains how the curated data can be used for analytics and Machine Learning. Finally, the book also covers advanced topics such as data migration, data sharing and emerging trends in the discipline of data engineering in GCP.

Data migration refers to the process of moving data between storage systems. Data migration can be done for multiple reasons, such as migrating data between two different database types, such as migrating MS SQL Server data to PostgreSQL. Data migration can also be done between a relational database, such as PostgreSQL, and a NoSQL database, such as MongoDB, as part of a legacy modernization initiative. In the modern cloud era, organizations are moving more and more applications from on-premise data centers to cloud environments. As part of these initiatives, the database is also migrated from an on-premise database to a cloud database. The cloud database can be either a database installed on a cloud virtual machine (infrastructure as a service) or a migration to a managed database service such as Cloud SQL (platform as a service). Cloud-managed database services are cloud native services that have a different architecture compared to an on-premise database server. Hence, there are additional considerations to keep in mind for such cloud database migrations.

After finishing this book, you will possess the technical competence to design, build, and monitor professional-grade data solutions on Google Cloud. You will be ready to tackle real-world challenges, from automating complex workflows to leveraging AI for predictive analytics in any enterprise environment.

What you will learn:
- Develop highly scalable, modern data engineering solutions in GCP.
- Optimize BigQuery performance using advanced table partitioning and data clustering.
- Build streaming pipelines using Pub/Sub, Dataflow, and the Apache Beam framework.
- Deploy Spark and Hadoop clusters on Dataproc with GCS lakes.
- Apply data mesh, Generative AI, and decentralized data strategies.
- Learn enterprise ETL and ELT architectures through managed Cloud Composer and Apache Airflow.

Who this book is for:
This book is for data architects, data engineers, data analysts, and ML engineers working on transforming raw data to curated, quality data for enterprise consumption. It caters to beginners as well as experienced data professionals and students who want to become data professionals.

Contents: