Concurrent Data Processing In Elixir
Download Concurrent Data Processing In Elixir PDF/ePub or read online books in Mobi eBooks. Click Download or Read Online button to get Concurrent Data Processing In Elixir book now. This website allows unlimited access to, at the time of writing, more than 1.5 million titles, including hundreds of thousands of titles in various foreign languages.
Concurrent Data Processing in Elixir
Author: Svilen Gospodinov
language: en
Publisher: The Pragmatic Programmers LLC
Release Date: 2021-07-25
Learn different ways of writing concurrent code in Elixir and increase your application's performance, without sacrificing scalability or fault-tolerance. Most projects benefit from running background tasks and processing data concurrently, but the world of OTP and various libraries can be challenging. Which Supervisor and what strategy to use? What about GenServer? Maybe you need back-pressure, but is GenStage, Flow, or Broadway a better choice? You will learn everything you need to know to answer these questions, start building highly concurrent applications in no time, and write code that's not only fast, but also resilient to errors and easy to scale. Whether you are building a high-frequency stock trading application or a consumer web app, you need to know how to leverage concurrency to build applications that are fast and efficient. Elixir and the OTP offer a range of powerful tools, and this guide will show you how to choose the best tool for each job, and use it effectively to quickly start building highly concurrent applications. Learn about Tasks, supervision trees, and the different types of Supervisors available to you. Understand why processes and process linking are the building blocks of concurrency in Elixir. Get comfortable with the OTP and use the GenServer behaviour to maintain process state for long-running jobs. Easily scale the number of running processes using the Registry. Handle large volumes of data and traffic spikes with GenStage, using back-pressure to your advantage. Create your first multi-stage data processing pipeline using producer, consumer, and producer-consumer stages. Process large collections with Flow, using MapReduce and more in parallel. Thanks to Broadway, you will see how easy it is to integrate with popular message broker systems, or even existing GenStage producers. Start building the high-performance and fault-tolerant applications Elixir is famous for today. What You Need: You'll need Elixir 1.9+ and Erlang/OTP 22+ installed on a Mac OS X, Linux, or Windows machine.
Concurrent Data Processing in Elixir
Learn different ways of writing concurrent code in Elixir and increase your application's performance, without sacrificing scalability or fault-tolerance. Most projects benefit from running background tasks and processing data concurrently, but the world of OTP and various libraries can be challenging. Which Supervisor and what strategy to use? What about GenServer? Maybe you need back-pressure, but is GenStage, Flow, or Broadway a better choice? You will learn everything you need to know to answer these questions, start building highly concurrent applications in no time, and write code that's not only fast, but also resilient to errors and easy to scale. Whether you are building a high-frequency stock trading application or a consumer web app, you need to know how to leverage concurrency to build applications that are fast and efficient. Elixir and the OTP offer a range of powerful tools, and this guide will show you how to choose the best tool for each job, and use it effectively to quickly start building highly concurrent applications. Learn about Tasks, supervision trees, and the different types of Supervisors available to you. Understand why processes and process linking are the building blocks of concurrency in Elixir. Get comfortable with the OTP and use the GenServer behaviour to maintain process state for long-running jobs. Easily scale the number of running processes using the Registry. Handle large volumes of data and traffic spikes with GenStage, using back-pressure to your advantage. Create your first multi-stage data processing pipeline using producer, consumer, and producer-consumer stages. Process large collections with Flow, using MapReduce and more in parallel. Thanks to Broadway, you will see how easy it is to integrate with popular message broker systems, or even existing GenStage producers. Start building the high-performance and fault-tolerant applications Elixir is famous for today. What You Need: You'll need Elixir 1.9+ and Erlang/OTP 22+ installed on a Mac OS X, Linux, or Windows machine.
Concurrent Data Pipelines with Broadway in Elixir
"Concurrent Data Pipelines with Broadway in Elixir" Unlock the full potential of data-intensive application design with "Concurrent Data Pipelines with Broadway in Elixir." This comprehensive guide takes readers from the foundational principles of concurrency and parallelism on the BEAM virtual machine, through the architectural intricacies of Broadway, Elixir’s premier data processing library. With detailed explorations of process isolation, message passing, and supervision trees, this book provides both the theoretical underpinnings and practical insights required to build robust, scalable, and fault-tolerant systems. It delves into GenStage’s influence on Broadway, guiding engineers to leverage Elixir’s unique strengths for event-driven workflows. Beyond the core architecture, the book offers an authoritative roadmap for integrating Broadway with diverse data sources—ranging from message queues like RabbitMQ and Kafka to custom APIs and streaming file inputs. Readers will master advanced processor patterns, batching strategies, and real-world output routing scenarios, ensuring consistent performance under heavy and unpredictable loads. Emphasis is placed on error handling, back-pressure mechanisms, transactional guarantees, and ensuring idempotency both at input and output stages—a must for reliable, production-grade data flows. Operational excellence is a central theme, with in-depth coverage of observability, telemetry, benchmarking, and reliability engineering. From scaling across distributed environments and implementing Kubernetes operator patterns, to safeguarding pipelines with modern security, compliance, and governance best practices, the book prepares you for the complexities of enterprise deployment. Concluding with advanced patterns, machine learning integration, and real-world case studies, "Concurrent Data Pipelines with Broadway in Elixir" is the definitive resource for any engineer or architect seeking to build, operate, and evolve next-generation data pipelines in Elixir.