MaestroRAG: Orchestrated Pipeline Architecture for Efficient RAG on Edge Devices

Abstract

Retrieval-Augmented Generation (RAG) offers a compact yet powerful alternative to massive Large Language Models (LLMs), making it attractive for privacy- and resource-constrained edge deployments. However, orchestrating RAG on devices with limited compute, memory, and power is nontrivial. We address this challenge with MaestroRAG, a fine-grained, three-stage pipeline architecture that systematically partitions encoding, retrieval, and generation across the CPU cores and a single GPU. Our design is driven by a detailed workload characterization that reveals how to best assign each RAG stage to CPU or GPU resources, coupled with an adaptive batching strategy to balance concurrency and memory usage. We further incorporate opportunistic caching to eliminate redundant lookups and computations. Implemented on three edge platforms, including an NVIDIA Jetson and consumer-grade GPUs, MaestroRAG reduces latency by up to 12× and improves throughput by 5.6× over EdgeRAG; against PipeRAG, the strongest baseline, the throughput gain is 1.35×. Energy per query is 3× lower than FlashRAG. By efficiently leveraging CPU resources for encoding and retrieval while reserving the GPU for generation, MaestroRAG lowers resource contention and power usage, bringing on-device RAG closer to real-world viability.