Performance evaluation and interpretability of a Qwen LangGraph agent for AI written and AI rewritten text detection

Abstract

The widespread adoption of large language models has increased the need to distinguish human-authored text from AI-generated content, including cases in which generated text is subsequently rewritten or paraphrased. In practice, many detectors output only a single AI-likelihood score or percentage, offering limited transparency and little indication of whether a passage appears directly AI-written (AIW) or AI-rewritten (AIR). This study presents an agentic detection framework for two connected tasks: binary provenance detection (Human versus AI) and conditional subtype detection (AIW versus AIR). The dataset combined the Human-ChatGPT Comparison Corpus (HC3) and the ai-text-detection-pile with 4,348 pre-generated AIR records produced by a single GPT-based rewriting workflow. The proposed Qwen–LangGraph AI agent was compared to a HashingVectorizer–TF-IDF–SGD classifier, DeBERTa-v3-xsmall, and ELECTRA-small. On the binary test set, DeBERTa-v3-xsmall achieved the best discrimination (99.4995% accuracy and 99.9911% AUROC), whereas the agent achieved 98.8816% accuracy and 99.9187% AUROC. On the AI-only test set (10,769 AIW; 1,274 AIR), the agent obtained the highest threshold-based accuracy (96.7367%), macro-F1 (90.0357%), and AIR F1 (81.8643%), but its AUROC (83.2507%) and AUPR (79.7336%) were lower than those of the DeBERTa-v3-xsmall and ELECTRA-small transformers. Paired DeLong and McNemar analyses with Holm correction showed that the agent improved threshold-based AIR performance over the lexical baseline. Interpretability was operationalised through lexical weights, linguistic and out-of-distribution diagnostics, sentence-level indicators, and route traces. The findings support the use of AI as decision support, not as automated proof of plagiarism or misconduct.