Prompt engineering a large language model with evidence-based persuasive features to improve confidence in mental health professionals: a pilot randomized experiment

Abstract

Background Depression carries a heavy global burden, yet treatment gaps persist largely because individuals lack confidence in mental health professionals. Psychoeducation can shift these beliefs, but scaling persuasive messages is difficult. Large language models (LLMs) offer a scalable solution, though the specific text-based features that make LLM-generated psychoeducation persuasive remain unidentified. This study identified these features and tested their integration into an LLM prompt to shift confidence in mental health professionals. Methods In Phase 1, 168 participants rated text pairs contrasting high versus low levels of four candidate features. In Phase 2, 40 participants were randomized to read psychoeducational passages generated by either a prompt incorporating the retained features (n = 20) or a baseline prompt (n = 20). The primary outcome was pre-to-post change in confidence in mental health professionals. Results Source credibility, argument quality, and processing fluency significantly boosted both perceived credibility and persuasiveness (all p < 0.01); affective warmth and empathy was excluded. Participants reading feature-engineered texts showed greater improvement in attitudes than the control group (t(38) = 3.37, p = 0.002, d = 1.07). Controlling for baseline scores, the group effect remained significant (F(1,37) = 9.31, p = 0.004, partial η2 = 0.20). Conclusion Strategically prompt-engineered LLM outputs incorporating empirically selected persuasive features significantly improve confidence in mental health professionals. This pilot study provides a preliminary evidence-based framework that may inform scalable, LLM-powered public mental health interventions.