A Lightweight Hybrid Framework for Detecting AI-Generated Scientific Text Using Multi-granularity Features and Language-Model Surprisal

Abstract

The increasing adoption of generative artificial intelligence (Gen-AI) in scientific writing has transformed scholarly communication while also raising concerns about authorship transparency and research integrity. The growing use of large language models (LLMs) has increased the need for methods that can screen AI-generated text while limiting false positives on human-written scientific prose. This study presents a lightweight hybrid framework that combines word-level TF–IDF features using 1–3-grams, character-level TF–IDF features using 3–6-grams, calibrated linear classification, and GPT-2-based language-model surprisal. The model outputs are combined through a weighted ensemble. Evaluation was conducted on a multi-corpus collection of 8,197 documents, comprising 4,100 human-written and 4,097 AI-generated documents, with a held-out evaluation set of 1,640 documents containing 820 documents from each class. The full ensemble achieved 95.80% accuracy, 97.10% precision, 94.35% recall, 95.71% F1-score, and 98.92% ROC-AUC. For the held-out human-written scientific documents, the reported specificity was 96.71%, corresponding to a false-positive rate of 3.29%. A separate five-fold stratified stability analysis on the complete corpus yielded a mean accuracy of 94.90% ± 0.53 and ROC-AUC of 98.58% ± 0.26. Within the evaluated corpus conditions, the findings demonstrate the potential of combining multi-granularity linguistic features and language-model surprisal for efficient scientific-text screening. The study provides a foundation for developing computationally efficient screening tools that can support editorial assessment, promote transparency in AI-assisted scientific writing, and contribute to research integrity while preserving human oversight.