WHO ASSURES THE ASSURER? A POSITION ON META-ASSURANCE FOR AI-MEDIATED EVALUATION

Abstract

Artificial intelligence is now used to evaluate other systems. Large language models (LLMs) generate assurance profiles, control questions and test cases, judge benchmark outputs, and support automated red teams. At the same time, the evaluated systems are often LLM-based agents themselves. In this position paper, we argue that this situation creates a new type of assurance problem. Three basic assumptions of classical evaluation are broken: (1) the evaluator and the target can fail in the same way (correlated blind spots); (2) the evaluation criteria are written by the same type of model that is evaluated (circular criteria); and (3) the target can influence the inputs of the evaluator (injectable evaluation). We connect this problem to known results in computer science and propose seven principles for what we call metaassurance. Finally, we present four testable propositions for future research.