Building High-Performance Streaming and Analytics Platforms

Abstract

The quantity of data that is created, processed, and streamed into a contemporary organisation is significant. As an engineering challenge, it is how to keep this up to date at all times and how to ingest, process, and serve this data with high availability and fault tolerance. This paper reviews architectural patterns, technology decisions, and optimization strategies of high-performance streaming and analytics systems. We cover Lambda and Kappa architecture, the messaging systems, essential stream processing engines, the available storage solutions, and best practices. Benchmark data and design trade-offs are then provided to guide practitioners on such systems, which allow millions of events per second with sub-10ms latency.