Speech-Based Human Emotion Recognition

Abstract

Emotion is a fundamental component of human communication, and the ability of machines to recognize it from speech has become increasingly important for applications such as virtual assistants, call-center analytics, mental health monitoring, and human-robot interaction. Traditional Speech Emotion Recognition (SER) systems have relied heavily on handcrafted acoustic features paired with classical machine learning classifiers, which often struggle to capture the complex temporal and spectral patterns present in emotional speech. This limitation motivates the exploration of deep learning architectures capable of modeling both the spatial and sequential characteristics of the speech signal. In this paper, we propose and evaluate a comparative SER framework built around three classification paradigms of increasing architectural complexity: a Support Vector Machine (SVM) with an RBF kernel as a classical baseline, a 2D Convolutional Neural Network (CNN) for spatial pattern extraction from spectrogram representations, and a Multiscale LSTM Fully Convolutional Network (MLSTM-FCN) designed to capture long-term prosodic contours and intonation dynamics. All models are trained and evaluated on the RAVDESS dataset across 8 discrete emotional categories — neutral, calm, happy, sad, angry, fearful, disgust, and surprised. Each raw audio sample undergoes silence trimming and amplitude normalization before a unified feature extraction stage that combines 13 to 40 Mel-Frequency Cepstral Coefficients (MFCCs) representing the vocal tract’s spectral envelope, 12 chroma bins capturing harmonic pitch class distribution, and 7 spectral contrast sub-bands measuring peak-to-valley frequency energy ratios, together forming a single standardized feature vector per sample. Experimental evaluation shows a clear performance progression across the three models: the SVM baseline achieves 88.5