Scalable ETL Pipeline Architectures for Real-Time Transaction Analytics: Bridging Data Engineering and Business Operations

Abstract

This review examines the architectural, technological and organisational conditions required to develop scalable extract, transform and load pipelines for real-time transaction analytics. Its purpose is to clarify how contemporary data-engineering capabilities can be aligned with operational decision-making across transaction-intensive enterprises. The study adopts a structured narrative review of scholarly and technical literature on batch, micro-batch, stream-processing, Lambda, Kappa, event-driven, cloud-native, lakehouse and serverless architectures, with additional attention to governance, security, observability, resilience and emerging-economy implementation contexts. The findings indicate that no single architectural model is universally optimal. Batch processing remains valuable for reconciliation, regulatory reporting and historical analysis, whereas stream-oriented and event-driven designs are better suited to fraud detection, payment monitoring, inventory visibility and other latency-sensitive operations. Hybrid architectures offer the strongest balance between speed, correctness, recoverability and cost. The review further finds that scalability depends not only on distributed computing, but also on partitioning, state management, change data capture, automated testing, lineage, data contracts, quality controls and service-level objectives. Organisational alignment, cross-functional ownership and regulatory compliance are equally decisive in determining whether technical capability produces measurable business value. The study concludes that real-time analytical performance must be evaluated through both engineering and operational outcomes. It recommends use-case-driven architectural selection, resilient hybrid deployment, embedded security and governance, automated quality assurance, transparent AI-assisted pipeline management and stronger collaboration between technical and business teams. Future research should develop standardised benchmarks combining latency, accuracy, resilience, cost, sustainability and operational impact, while giving greater attention to infrastructure-constrained and emerging-economy environments. These priorities are essential for building data infrastructures capable of supporting responsive, evidence-based enterprise operations at scale.