Understanding Cross‑Lingual Model Behavior in Mental Health NLP: Evidence from English–Bengali Transformer Models

Abstract

Abstract Understanding how machine learning models interpret mental health expressions across languages is essential for developing reliable, inclusive computational mental health systems. While transformer-based models have shown strong performance in English-language mental health detection, their interpretability and prediction behavior across languages with different linguistic characteristics remain insufficiently understood. This study examines cross-lingual interpretability in mental health natural language processing by analyzing how transformer models process and classify mental health expressions in English and Bengali social media text. To support this investigation, we introduce BMHD, a bilingual English-Bengali mental health dataset comprising 10,672 parallel text samples per language across 10 mental health classes and one neutral class, constructed through systematic data collection, translation, annotation, and human-evaluated validation. Multiple monolingual and multilingual transformer-based models are evaluated under English-only, Bengali-only, and bilingual settings. Beyond overall performance metrics, the analysis focuses on class-wise behavior, token-length effects, cross-lingual error patterns, attribution consistency, and agreement between multiple interpretability methods to explore differences in model interpretation across languages. The results reveal consistent cross-lingual disparities in prediction behavior despite semantic alignment between parallel texts. Models show reduced sensitivity to high-risk suicidal expressions and psychosis-related cues in Bengali, increased confusion between closely related mental health categories such as depression and suicidal thoughts, greater misclassification for a small number of unusually long sequences in the Bengali setting, and consistently low agreement among interpretability methods, particularly for Bengali predictions. These findings indicate that linguistic structure, translation effects, and representational imbalances between languages significantly shape how models interpret mental health signals, affecting both accuracy and interpretability. Overall, this study demonstrates that cross-lingual mental health classification cannot be treated as language-agnostic and highlights the need for linguistically aware evaluation frameworks. The dataset and empirical analyses presented in this work provide a valuable benchmark for future research on interpretable and responsible multilingual NLP for mental health, especially for low-resource languages such as Bengali.