Real-Time Data Anomaly Detection Architecture for Business and Service Metrics Using Autoencoder Model with a LocalStack-Simulated Cloud-Native Infrastructure

Abstract

Organizations utilizing distributed digital services generate ongoing operational metrics that can lead to failures and breaches of service level agreements if not watched proactively. In this paper, we discuss the design, implementation, and verification of a complete system for the near real-time detection of anomalies within business and service metrics. The system supports multiple tenants using separate client-specific LSTM autoencoder models and a cloud-native serverless architecture using AWS services, including Amazon DynamoDB, two AWS Lambda functions, Amazon EventBridge with scheduling rules, Amazon Simple Notification Service, and Amazon Simple Queue Service. The AWS services are provisioned and managed via Terraform, and the complete system is emulated locally with LocalStack. A Spring Boot application manages the validation of resources, the ingestion API, and the alerts. A supporting Python worker renders LSTM autoencoder inference for records aggregated in DynamoDB. The system provides a 0.91 precision and 0.88 recall against simulated workloads with multiple clients. The LocalStack-based setup, during the development and testing phases, was useful for quickly iterating and debugging distributed workflows without incurring cloud costs. Terraform-based Infrastructure as Code (IaC) enabled the system to provision resources and achieve alerts in under 500 milliseconds.