A Reproducible and Budget-Aware Framework for Evaluating Local Retrieval-Augmented Generation (RAG) Robustness to Perturbations

Abstract

The idea of context is no longer considered secondary in the construction of language-model systems. With the use of local Retrieval-Augmented Generation, even a tiny modification of the prompt or the context might produce another set of retrievals, citations, and ultimately different answers; however, in practice, tests are often performed with only one version of the question. In this work, we suggest a local context-engineering framework for exploring perturbation robustness, reproducibility, and budgetfriendly assessment in one unified pipeline. Local RAG is built on several small teaching packs for the models; perturbations are introduced to the queries, each experiment is recorded in capsule format, and a gate mechanism based on the decision tree is used to judge if the whole perturbation suite can be skipped, minimized, or run. As a result, the augmented capsule-derived data set contains 3,570 perturbation rows, out of which 2,619 labeled rows are used for retraining. In this labeled subset, BLEU scores are available in 446 cases, answer perplexity and semanticjudgment scores are available in 1,668 cases, citation overlap is available for all rows, and retrieval overlap is available for 2,099 rows. The final decision tree obtains 0.915 ROC-AUC, 0.891 PR-AUC, 0.913 accuracy, 0.863 F1 score, and expected savings of 76.5% on the held-out augmented test split. A more conservative threshold setting lowers the stable-risk value from 0.084 to 0.062, while reducing expected savings to 57.4%. Thus, we provide a local study pipeline to examine the effect of perturbations on answer stability, contradictions, robustness, and budget-friendly evaluation.