A Computer Vision Approach for Non-contact Psychophysiological Assessment: Speaking-Aware Video-Based Stress Detection Using Extended TSST Protocol

Abstract

The assessment of psychological stress through non-contact computer vision methods faces significant challenges when facial expressions are affected by motions caused by speech. In this paper, we introduce a novel video-based stress detection system with speaking awareness that dynamically processes facial features according to online detection of speech activities. The system employs an extended 41-minute Trier Social Stress Test protocol with a controlled baseline, moderate stress induction, peak stress increase, and recovery stages to comprehensively record the dynamics of stress development. Facial landmark detection is handled by MediaPipe Face Mesh and provides 468 three-dimensional landmarks with sub-pixel accuracy, while the speaking-aware processing strategy dynamically selects the most appropriate feature sets: seven robust ones during speech-containing intervals like face movement, eye aspect ratio, and forehead wrinkles, and nine full-featured ones during silent intervals with additional mouth and jaw metrics. The adaptive strategy addresses the intrinsic limitation of traditional facial analysis, which treats speaking and non-speaking states homogeneously. Deployment on the Raspberry Pi CM4 edge computing device enables real-time operation with privacy preservation via localized processing. Empirical testing with 71 participants illustrates strong performance with an F1-score of 83.16%, outperforming conventional methods by 4.24%. Cross-population validation affirms good generalization potential across various populations for the entire 41-minute protocol, thereby demonstrating the system's applicability to real-world applications in stress tracking across healthcare, educational, and workplace well-being contexts.