How different judgment strategies affect accuracy in generative AI text discrimination: BERT-based clustering and hierarchical Bayesian logistic regression

Abstract

Large language models (LLMs)-generated texts become increasingly similar to human writing. The purpose of this study was to examine (1) how individuals distinguish between human-written and LLM-generated texts, focusing on their judgment strategies; (2) the relationship between judgment strategies and detection accuracy; and (3) the interaction effects of gender, age, and frequency of large language model (LLM) use. Participants were asked to judge whether the given texts (e.g., public comments) were written by LLMs or humans and to explain their reasoning using open-ended questions. The public comments were written by humans and generated using seven LLMs (ChatGPT [4o and o1], Claude3.5, Gemini, Microsoft Copilot, Llama3.1, and Perplexity). First, we classified participants’ explanations of reasoning related to judgments into types using BERT-based clustering (i.e., Sentence-BERT and hierarchical cluster analysis). BERT-based clustering suggested four interpretable clusters. Based on their characteristic patterns, we interpreted these clusters as four hypothetical judgment strategy types and tentatively proposed that they form a conceptual continuum: (1) Unsubstantiated intuition, (2) Stylistic-Cue, (3) Syntactic-Cue, and (4) Higher-Order Pattern Recognition (HOPR). A hierarchical Bayesian logistic regression analysis revealed that the detection accuracy varied across judgment types, with the HOPR type achieving relatively higher performance than the other strategy types, although its detection accuracy remained near the chance level. Additionally, we identified interaction effects between strategy type and gender; only female participants who used the HOPR strategy showed lower detection accuracy than male participants, although the qualitative interpretation of this difference warrants further empirical investigation. These findings emphasize the role of judgment strategies and individual differences in discriminating between human and LLM-generated content.