Efficient training-free query routing framework for federated retrieval-augmented generation

Abstract

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by grounding responses in external documents retrieved at inference time. In federated RAG (FedRAG), documents are distributed across multiple autonomous corpora that cannot be centrally aggregated due to privacy, regulatory, or organizational constraints. A key challenge in FedRAG is query routing: selecting a subset of relevant corpora for each query without broadcasting to all participants. Existing routing approaches rely on supervised learning, requiring a uniform retrieval embedding model across corpora and costly retraining whenever the federation changes. These assumptions violate corpus autonomy and limit scalability in dynamic federated environments. We propose RAGSpot , a training-free query routing framework that enables efficient corpus selection without requiring a shared local retrieval embedding model. RAGSpot represents each corpus using compact cluster-level metadata derived from its vector index and estimates corpus relevance via query-to-cluster matching. In addition, it introduces a rank-based fusion strategy that jointly leverages document-level and corpus-level rankings to support robust aggregation across heterogeneous embedding models. Experiments with corpora from the BEIR benchmark show that RAGSpot improves retrieval accuracy over the strongest training-based router by 1.23– \(1.48\times\) in nDCG and 1.28– \(1.66\times\) in Recall, and over the strongest LLM-based router by 1.63– \(2.01\times\) in nDCG and 1.93– \(2.34\times\) in Recall, without any further training. Also, RAGSpot achieves query latency that is comparable to, or even lower than, that of SOTA methods on average.