Emotion Detection using Speech and Text – Based Sentiment Analysis: A Hybrid Multimodal Application

Abstract

Detecting human emotions accurately remains a challenging problem in affective computing, particularly when systems rely on a single system. This paper presents a hybrid approach that combines speech-based acoustic features with text-based sentiment analysis to improve emotion detection. On the speech side, we trained a Conv1D convolutional neural network on the RAVDESS dataset using 1,440 audio recordings expanded to 5,760 samples through augmentation, achieving 77% accuracy and a macro-F1 of 0.76 across eight emotion classes. For text, we used a pretrained DistilRoBERTa transformer on the dair ai/emotion Twitter dataset, which classified samples with over 99% confidence across joy, fear, sadness, and anger. We also compared two rule-based lexicon methods—VADER and TextBlob—on 500 test samples, where VADER achieved roughly 80% directional alignment and TextBlob about 75%. The key contribution of this work is a confidence-based fusion strategy that combines both modalities on a balanced 5-class test set of 250 paired speech–text samples, reaching 98%accuracy, 98.03%precision, and a 98.01%F1-score. This is a 21-point improvement over the speech-only model and 18–23 points over the lexicon baselines. Our per-class analysis shows that fusion helps most with low-arousal emotions like sadness, where the F1-score rose from 0.69 (speech alone) to 0.97.