Towards Automatic ACT-Based Analysis of Chats

Abstract

Affect Control Theory (ACT) models social interaction through culturally shared affective meanings, yet applying it to naturalistic dialogue requires manual event annotation whose inter-coder reliability has never been benchmarked. We report the first multi-coder reliability study for ACT event annotation, in which three trained experts independently coded 49 German exam-anxiety counseling chats. Experts varied substantially in annotation granularity (19–30 distinct behaviors, 26.0–38.5 events per chat) and produced low-to-moderate deflection agreement (pairwise r = −0.016 to 0.213). To assess automation feasibility, we developed an open-weight LLM agent (Qwen2.5-32B, local via Ollama) performing the full ACT pipeline. Across 10 runs the agent achieved high self-consistency (deflection ICC = 0.785, behavior Jaccard = 0.850). The agent’s strongest expert correlation (r = 0.392 with Expert 3) was the highest of any annotator pair, demonstrating that open-weight automation is feasible within the range of human agreement.