A new latent Dirichlet allocation for enhancing sentiment analysis in social network

Abstract

The volume of sentiment-rich text data produced on a daily basis by social network sites such as Twitter is very large. However, traditional approaches to sentiment analysis, especially approaches based on the Bag-of-Words (BoW) model have a difficult time modeling more subtle thematic content structure and interrelationships in this data. The approaches tend to narrow down text to an easy word frequency level, leading to the loss of valuable semantic information required for accurate sentiment classification. This problem is even more evident when one has to work with large complex data. Furthermore, being high-dimensional feature spaces, BoW methods produce represented spaces that can saturate traditional classification models, decreasing their level of accuracy and reliability. The article overcomes these drawbacks of traditional topics modeling by considering using Latent Dirichlet Allocation (LDA) as an intricate subject topic extraction method in English-language data tweets sentiment examination. LDA can reveal latent thematic structures in the tweets that enhance representation of features to be used in sentiment classification. The research methodology will consist of preprocessing the Twitter Sanders file, creating feature sets of different granularity (E40, E100, E300, E500), and testing them in Decision Trees (DT), Naive Bayes (NB) and K-Nearest Neighbors (KNN). Such distinction of sets of features allows investigating the issue of influence of feature granularity on the performance of a classifier in more detail. Findings indicate that LDA-based features substantially outperform BoW-based features in sentiment classification across all evaluation metrics. Decision Tree and Naive Bayes classifiers achieved the highest accuracy, precision, and recall using LDA features, with optimal performance observed at the E300 configuration. KNN behaved better with the smaller feature sets and poorly with the larger features sets. These improvements have also been confirmed using statistical tests and this includes paired t-tests, Wilcoxon signed-rank tests and are shown to be statistically significant across the board. These findings establish LDA as an effective and computationally efficient feature extraction technique for sentiment classification in social media data. This study contributes a systematic methodology for enhancing sentiment analysis accuracy and provides a foundation for future work integrating LDA with advanced machine learning algorithms.