A multi-model LLM framework for AI-assisted psychological scale development: initial psychometric evaluation of the Emotion Regulation Scale (ERS)

Abstract

Emotion regulation is a key construct in mental health, yet existing measures differ in conceptual coverage, with some focusing on specific strategies or regulatory difficulties rather than broader cognitive, behavioral, and contextual regulation processes. This study aimed to develop an Emotion Regulation Scale (ERS) grounded in Gross’s Process Model, using large language models (LLMs) as assistive tools within an established psychometric framework. This cross-sectional study included 743 adults recruited through convenience sampling. Participants were randomly divided into two non-overlapping subsamples, one used for exploratory factor analysis and the other for confirmatory factor analysis. Multiple LLMs and AI-based tools assisted with item generation, refinement and bias screening, translation, and post-EFA construct interpretation, with human oversight throughout development. An initial pool of 25 items was generated and refined to produce the final 17-item ERS. The ERS revealed a preliminary four-factor structure, with EFA explaining 50.85% of the variance. CFA provided preliminary support for a post hoc refined 17-item model (χ²/df = 1.95, CFI = 0.934, TLI = 0.918, IFI = 0.935, RMSEA = 0.050). Reliability was acceptable for early-stage scale development (α = 0.68–0.74; ω = 0.68–0.75). Convergent validity showed positive correlations between adaptive strategies and cognitive reappraisal ( r = 0.522–0.608) and between maladaptive cognitions and suppression ( r = 0.400). Concurrent validity showed associations with depression ( r = 0.432), anxiety ( r = 0.462), well-being ( r = -0.330), and quality of life (QOL1 = -0.357; QOL2 = -0.295), while adaptive strategies correlated with depression ( r = -0.184 to -0.239), anxiety ( r = -0.141 to -0.211), well-being ( r = 0.150 to 0.203), and quality of life ( r = 0.211 to 0.375). Known-groups validity indicated significant differences across depression and anxiety levels (MCR: F(4, 738) = 39.32; 64.49, p < 0.001). The findings provide preliminary psychometric evidence for the 17-item ERS and suggest that it may complement existing measures by capturing broader cognitive, behavioral, and contextual emotion regulation processes. The study illustrates the potential for LLM-assisted scale development within established psychometric procedures and human oversight. Further research using diverse samples is needed to evaluate the scale’s psychometric properties, generalizability, and broader utility.