Deletion Isn't Enough: Auditing RAG for Selective Forgetting

Abstract

For information access systems, it is not enough for outputs or answers to be relevant or correct; they must also be permitted. This paper highlights a research gap: non-disclosure obligations often concern propositions that must not be stated, while deployed Retrieval-Augmented Generation (RAG) systems enforce restrictions through record-level handling such as document removal or access-control lists. Such systems can appear compliant while still disclosing revoked facts in generated answers. Most RAG evaluations fail to assess this important aspect of compliance. We explore the nature of this gap and introduce Forgetting-by-Design (FBD), a mechanism-agnostic audit that runs probes across paired system states before and after revocation. FBD separates compliance into two observable channels---retrieval/citation exposure and answer-level disclosure or abstention ---and reports the cost of compliance using matched lawful controls and substitution-aware utility signals. We instantiate FBD in a reproducible RAG setting and show how the resulting report card reveals failures that single metrics miss: retrieval exposure can be suppressed while answer-level leakage persists, and interventions that reduce disclosure can still degrade lawful utility or collapse citation coverage.