HLLC: A Four-Stage Human-AI Collaborative Annotation Framework for Panic Emotion Recognition in Social Networks

Abstract

Public emergencies often trigger large-scale panic emotions, which are further amplified by social media, potentially leading to serious consequences such as resource hoarding and social disorder. Therefore, timely identification and intervention of panic are of great significance. However, the development of panic emotion recognition models is severely constrained by the scarcity of high-quality annotated datasets. Existing coarse-grained sentiment classification fails to distinguish panic from fear and anxiety, where the latter two are characterized by high arousal and a sense of loss of control. Furthermore, purely manual annotation is expensive and hard to scale, while fully automated annotation suffers from model hallucinations and conservative bias. To address the challenges in constructing large-scale panic emotion datasets, this paper leverages large language models (LLMs) and proposes a four-stage progressive human-AI collaborative annotation framework named HLLC (Human-LLM Collaborative Labeling). The framework sequentially performs data sampling, LLM initial labeling, crowdsourced verification, and model generalization. Through representative sampling, multi-model consistency comparison for zero-shot labeling, confidence-based sample routing, and BERT fine-tuning, the framework balances annotation quality and efficiency. Experiments on the CrisisNLP Hurricane Sandy dataset [1] show that the HLLC framework achieves a panic F1 score of 0.787 and an overall accuracy of 0.857, improving by 33.3% over lexicon methods and boosting panic recall by 78.7% over pure LLM annotation. These results demonstrate that HLLC enables large-scale, fine-grained panic emotion dataset construction at low cost, offering a practical solution for emergency public opinion monitoring and early warning.