Analysis of the Recall Crisis and Semantic Robustness of Llm Detectors Under Hierarchical Deep Adversarial Rewriting Attacks

Abstract

Current LLM detectors remain vulnerable to adversarial attacks because they focus on superficial statistics of the text. In this study, the robustness of state-of-the-art (SOTA) models was tested on a hierarchical attack dataset of 24,000 examples generated using T5, BART, and PEGASUS, based on the PAN 2025 dataset. Analyses revealed that the DeBERTa-v3 model experienced a “Recall Crisis” in the deep rewriting (V4) scenario, with its Recall dropping dramatically to 55%. In contrast, the proposed SBERT-based semantic approach maintained an 80% success rate in the same scenario, unaffected by changes in word structure. The findings empirically demonstrate that semantic consistency, rather than statistical memorization, must be prioritized for reliable AI detection.