Modern Data Engineering with Apache Spark: A Hands-On Guide for Building Mission-Critical Streaming
Scott Haines
585 pages, parution le 22/03/2022
Résumé
Apache Spark applications solve a wide range of data problems from traditional data loading and processing to rich SQL-based analysis as well as complex machine learning workloads and even near real-time processing of streaming data. Spark fits well as a central foundation for any data engineering workload. This book will teach you to write interactive Spark applications using Apache Zeppelin notebooks, write and compile reusable applications and modules, and fully test both batch and streaming. You will also learn to containerize your applications using Docker and run and deploy your Spark applications using a variety of tools such as Apache Airflow, Docker and Kubernetes.
Reading this book will empower you to take advantage of Apache Spark to optimize your data pipelines and teach you to craft modular and testable Spark applications. You will create and deploy mission-critical streaming spark applications in a low-stress environment that paves the way for your own path to production.
What You Will Learn
- Simplify data transformation with Spark Pipelines and Spark SQL
- Bridge data engineering with machine learning
- Architect modular data pipeline applications
- Build reusable application components and libraries
- Containerize your Spark applications for consistency and reliability
- Use Docker and Kubernetes to deploy your Spark applications
- Speed up application experimentation using Apache Zeppelin and Docker
- Understand serializable structured data and data contracts
- Harness effective strategies for optimizing data in your data lakes
- Build end-to-end Spark structured streaming applications using Redis and Apache Kafka
- Embrace testing for your batch and streaming applications
- Deploy and monitor your Spark applications
Who This Book Is For
Professional software engineers who want to take their current skills and apply them to new and exciting opportunities within the data ecosystem, practicing data engineers who are looking for a guiding light while traversing the many challenges of moving from batch to streaming modes, data architects who wish to provide clear and concise direction for how best to harness and use Apache Spark within their organization, and those interested in the ins and outs of becoming a modern data engineer in today's fast-paced and data-hungry world
Part II. The Streaming Pipeline Ecosystem 8. Workflow Orchestration with Apache Airflow9. A Gentle Introduction to Stream Processing10. Patterns for Writing Structured Streaming Applications11. Apache Kafka & Spark Structured Streaming12. Analytical Processing & Insights
Part III. Advanced Techniques 13. Advanced Analytics with Spark Stateful Structured Streaming14. Deploying Mission Critical Spark Applications on Spark Standalone15. Deploying Mission Critical Spark Applications on Kubernetes
Prior to Twilio, Scott worked writing the backend Java APIs for Yahoo Games as well as the real-time game ranking and ratings engine (built on Storm) to provide personalized recommendations and page views for 10 million customers. He finished his tenure at Yahoo working for Flurry Analytics where he wrote the alerts and notifications system for mobile devices.
Caractéristiques techniques
| PAPIER | |
| Éditeur(s) | Apress |
| Auteur(s) | Scott Haines |
| Parution | 22/03/2022 |
| Nb. de pages | 585 |
| EAN13 | 9781484274514 |
Avantages Eyrolles.com
Livraison à partir de 0,01 € en France métropolitaine
Paiement en ligne SÉCURISÉ
Livraison dans le monde
Retour sous 15 jours
+ d'un million et demi de livres disponibles
Consultez aussi
- Les meilleures ventes en Graphisme & Photo
- Les meilleures ventes en Informatique
- Les meilleures ventes en Construction
- Les meilleures ventes en Entreprise & Droit
- Les meilleures ventes en Sciences
- Les meilleures ventes en Littérature
- Les meilleures ventes en Arts & Loisirs
- Les meilleures ventes en Vie pratique
- Les meilleures ventes en Voyage et Tourisme
- Les meilleures ventes en BD et Jeunesse
