Reliability Evaluation of Large Language Models for Social Media Sentiment Annotation: An Empirical Study Based on Model Agreement and Downstream Tasks

Abstract

Driven by booming social media and user-generated texts, sentiment analysis stands as a core natural language processing task, yet large-scale high-quality data labeling comes with steep costs and practical barriers. This work assesses how dependable large language models are for sentiment tagging, alongside how their labeled outputs shape subsequent classification effects. We build a dataset containing 1,543 Chinese entertainment comment snippets scraped from Bilibili. Under unified prompting, three LLMs—DeepSeek, Qwen and Doubao—produce zero-shot sentiment tags, while 500 sampled entries receive manual annotation to form authoritative benchmark labels. Cohen’s Kappa is adopted to quantify human-model annotation consistency, and TF-IDF features are input to Logistic Regression, Linear SVM and Random Forest for downstream classification evaluation. Among the three models, Doubao achieves the highest human-label consistency with a κ value of 0.800, exceeding Qwen (κ=0.746) and DeepSeek (κ=0.636). Classifiers trained on Doubao’s labels obtain optimal Macro-F1 values of 0.8125 (SVM) and 0.8174 (Random Forest). Obvious performance discrepancies exist across LLMs; high-quality annotations significantly boost downstream classification accuracy, which highlights the importance of selecting competent LLMs for sentiment labeling tasks.