Temporal Metamorphic Testing for RAG-Based LLMs under Evolving Knowledge

Abstract

Evaluating the temporal consistency of RetrievalAugmented Generation (RAG) systems is challenging because conventional evaluation methods rely on ground-truth answers, which are difficult to define under continuously evolving knowledge. This paper presents a temporal metamorphic testing framework for RAG-based large language models (LLMs) that evaluates relational consistency across multiple executions over time rather than comparing outputs against static reference answers. To support temporal metamorphic testing, we construct controlled temporal revision families from seed facts extracted from real-world documents. Each fact is systematically cloned and transformed across multiple timestamps to simulate evolving knowledge states. Based on this design, temporal metamorphic relations are instantiated to systematically test RAG systems under different knowledge snapshots. Experiments on 2344 metamorphic test instances show that 48.04% of them violate the expected temporal relations, indicating that temporal inconsistencies are widespread in the evaluated RAG system. These results demonstrate that temporal metamorphic testing provides an effective way to assess temporal reliability in RAG systems without requiring explicit ground-truth answers.