Leveraging Fine-tuned IndoBERT and Zero-shot GPT-4o for Mental Health Monitoring from Indonesian Tweets

Abstract

Social media provides valuable linguistic signals for population-level mental health monitoring, but its informal, ambiguous, and context-dependent language complicates automated classification. This study compares fine-tuned IndoBERT and zero-shot GPT-4o on Indonesian-language tweets across three tasks: seven-class emotion classification, four-class sentiment analysis, and binary suicidal ideation detection. A dataset of 4,280 tweets collected through keyword-based scraping from 2023 to 2025 was labeled using an AI-assisted annotation procedure, with a subset manually reviewed to ensure contextual relevance and label alignment. Of the total dataset, 3,424 tweets were used to fine-tune IndoBERT, while 856 held-out tweets served as a common evaluation benchmark. GPT-4o processed 850 test samples, with six inputs blocked by its built-in content moderation mechanism. The results show a task-dependent performance pattern. GPT-4o outperformed IndoBERT in emotion classification (Macro F1: 0.650 vs. 0.578), whereas IndoBERT performed better in sentiment analysis (0.570 vs. 0.517) and suicidal ideation detection (0.693 vs. 0.441). Despite comparable overall accuracy in suicidal ideation detection, GPT-4o achieved a recall of only 0.01 for the suicidal class, compared with 0.63 for IndoBERT, indicating a critical false-negative risk. These findings suggest that zero-shot GPT-4o performs well on prototypical emotional expressions, whereas task-specific IndoBERT fine-tuning provides more reliable performance for ambiguous sentiment and implicit suicidal ideation. Accordingly, the proposed framework provides a foundation for human-supervised, real-time, granular mental health monitoring using Indonesian tweets.