This work introduces a metamorphic testing framework that evaluates the consistency of RAG systems under corpus evolution, formalising a fault taxonomy and 11 mutation operators that systematically perturb the system at both the pre-chunk (retrieval index) and post-chunk (retrieved context) levels.
Abstract
Retrieval-Augmented Generation (RAG)-based LLM systems rely on external document corpora that can evolve and change over time. However, current evaluation methodologies (e.g., RAGAS) assess correctness against static snapshots, failing to detect faults when routine updates, factual changes, or noise alter the underlying data. We introduce a metamorphic testing framework that evaluates the consistency of RAG systems under corpus evolution. We formalise a fault taxonomy and 11 mutation operators that systematically perturb the system at both the pre-chunk (retrieval index) and post-chunk (retrieved context) levels. An empirical evaluation across five datasets and over 28k mutants reveals metamorphic violation rates of 4.9-10.2%. In a meta-evaluation against ground truth, our metamorphic oracle achieves F1 scores of 0.927-1.000, while the best RAGAS metric reaches only 0.570. Finally, we provide actionable insights into mitigating these faults through retrieval re-configuration, generator upgrades, and LLM-based reranking.
Evaluating the temporal consistency of RetrievalAugmented Generation (RAG) systems is challenging because conventional evaluation methods rely on ground-truth answers, which are difficult to define under continuously evolving knowledge. This paper presents a temporal metamorphic testing framework for RAG-based large language models (LLMs) that evaluates relational consistency across multiple executions over time rather than comparing outputs against static reference answers. To support temporal metamorphic testing, we construct controlled temporal revision families from seed facts extracted from real-world documents. Each fact is systematically cloned and transformed across multiple timestamps to simulate evolving knowledge states. Based on this design, temporal metamorphic relations are instantiated to systematically test RAG systems under different knowledge snapshots. Experiments on 2344 metamorphic test instances show that 48.04% of them violate the expected temporal relations, indicating that temporal inconsistencies are widespread in the evaluated RAG system. These results demonstrate that temporal metamorphic testing provides an effective way to assess temporal reliability in RAG systems without requiring explicit ground-truth answers.
Hongyu Wang, Dave Towey, Ge Zhang et al.· Annual International Compute...· 0 citations
Retrieval-Augmented Generation (RAG)-based systems\footnote{For brevity, RAG-based systems are referred to as RAG systems throughout this paper.} are increasingly deployed in high-stakes settings where correct behaviour depends not only on the language model but also on the retrieval component that selects external documents at inference time. While existing RAG evaluation metrics assess retrieval and generation quality on a per-query basis, typically relying on query-level test oracles such as reference answers or relevance annotations, they provide limited insight into whether a test suite adequately exercises the retrieval behaviour of the system as a whole. In this paper, we introduce Chunk Coverage (CC), an oracle-independent test adequacy criterion for testing the retrieval component of RAG systems. CC measures the fraction of corpus chunks that are retrieved at least once across a test suite, providing a structural view of which parts of the retrieval space have been exercised. We further show how CC can be used to guide test selection and generation by prioritising queries that expand coverage of previously unexercised retrieval regions. We evaluate CC on clinical and financial RAG system scenarios. CC-guided testing reaches 50% of attainable coverage 1.7x faster than random selection and 4.2x faster than redundancy-biased strategies. Moreover, CC improves fault detection effectiveness (APFD) by 10% to 25% over random, indicating earlier discovery of distinct retrieval faults. These results show that CC captures retrieval diversity relevant to effective testing without requiring test oracles.
Jinhan Kim, Samuele Pasini, Paolo Tonella· 1 citation
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to use external and domain-specific knowledge, but its reliability depends on the interaction between the generative model, embedding model, retrieval mechanism, and prompt construction strategy. We present RagTester, an automated end-to-end testing approach for RAG systems. RagTester generates retrieval documents, test inputs, and expected outputs; executes the tests; and evaluates the resulting answers using an LLM as a judge. Its test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria. We evaluate RagTester using eight LLMs and six embedding models, yielding 24 compatible configurations, and compare it with a baseline test-input generator. Across 72,000 test executions, RagTester detected 21,633 failures, 6.6% more than the baseline, and outperformed it in 20 of the 24 configurations. The detected failures include inaccurate retrieval, unsupported answers, incomplete use of retrieved context, and difficulties interpreting complex passages. These results show that coverage-oriented test generation can effectively expose failures caused by the interaction between retrieval and generation components and support the assessment of RAG configurations before deployment.
Ange Maiztegi, J. Ayerdi, Miren Illarramendi et al.· 0 citations
Software testing often depends on a test oracle to determine whether or not an output is correct. However, many real-world systems lack reliable or affordable oracles. Metamorphic testing (MT) alleviates this problem by using metamorphic relations (MRs), which examine the correctness of relations between outputs produced from related inputs across multiple executions. While the effectiveness of MT depends heavily on the quality of MRs, existing MR evaluation is typically guided by informal or study-specific criteria, limiting standardization and repeatability, especially for complex and AI-integrated systems. This paper proposes a structured MR-evaluation rubric that operationalizes key criteria to support consistent and repeatable assessment. We conducted an empirical study on nine systems under test (SUTs) spanning three levels of complexity, using fresh sets of LLM-generated MRs for each SUT. We then compared evaluations from human experts against two widely used large language models (LLMs), namely ChatGPT and Gemini, by configuring these models as MR evaluators using the same rubric. The results show that human and LLM evaluators are more consistent when assessing simpler SUTs, but their judgments increasingly diverge as system complexity grows and evaluation criteria become more varied. The study further reveals clear differences in LLM-evaluation behaviors, indicating that evaluator choice can affect MR assessment outcomes in complex settings. Overall, the proposed rubric provides a standard framework for MR evaluation, and supports a systematic comparison between human and LLM evaluators, offering empirical evidence on when LLM-based MR evaluation is more reliable, and where additional care is needed as SUT complexity increases.
Yifan Zhang, Dave Towey, Matthew Pike et al.· Annual International Compute...· 0 citations
This work introduces a search-based approach that identifies and evolves a set of natural language transformation rules with strong downstream effects on coding performance, and proposes DUALFIX, a staged repair pipeline that combines the evolved transformation rules with execution-feedback repair, addressing both specification-level and implementation-level failures.
Amal Akli, Melissa Akli, Cedric Richter et al.· 0 citations
Recent document parsers achieve table TEDS scores above 93 on OmniDocBench v1.6, yet community feedback and our audit reveal persistent failures on complex real-world tables. To quantify this gap, we introduce TableParseMap, a diagnostic benchmark of 916 real-world tables organized into five challenging scenarios and nine failure types. The strongest evaluated parser achieves only 85.03 TEDS, showing that aggregate benchmark scores conceal substantial weaknesses. Our analysis attributes these failures to three complementary limitations: large tables exceed the reliable processing scale of a single pass, weak or ambiguous visual cues hinder structure perception, and the reconstructed table may remain visually inconsistent with the image. We therefore propose DEC (Decompose--Enhance--Correct), a visual-consistency-guided agentic framework that improves frozen table parsers without retraining. DEC uses a general VLM as the controller: Decompose partitions large tables along structure-aware boundaries, Enhance exposes weak visual evidence and reparses transformed views, and Correct diagnoses and repairs residual errors. A Visual Consistency Gate (VC-Gate) selectively triggers intervention, while a Visual Consistency Ranker (VC-Ranker) verifies candidate updates and supports rollback without ground-truth HTML at inference time. We further derive a 1,977-table Consensus-Hard Set from 4,556 candidates through offline metrics and cross-model consensus. Across three frozen parsers, DEC improves TEDS by 1.57 points on average; on TableParseMap, gains reach 1.89 points overall, 2.62 on structural errors, and 5.66 on large tables.
Jutao Xiao, Yuan Qu, Dongsheng Ma et al.· 0 citations