Comparison of SMOTE, Class Weighting, and Classical Machine Learning Models on the ID-SMSA Indonesian Stock Market Dataset
TL;DR
A transparent and reproducible classical baseline is provided that situates transformer-based and deep-learning approaches on ID-SMSA within a well-defined reference frame and confirms that linear SVM on TF-IDF is robust to moderate imbalance at IR = 2.41.
Abstract
Sentiment classification of social-media text related to the Indonesian stock market is a growing research area. The ID-SMSA dataset is the publicly available labelled corpus for this domain, yet class-imbalance handling strategies on this dataset have not been systematically compared across multiple classifiers. This paper evaluates Multinomial Naive Bayes, linear Support Vector Machine (SVM), and Random Forest under three imbalance-handling conditions: no handling, class weighting, and SMOTE. All experiments use the full 3,287-tweet dataset with an 80:20 stratified split and report macro F1 as the primary metric. SMOTE consistently improves macro F1 across all classifiers. The largest gain is on Naive Bayes (+0.137, from 0.589 to 0.726). The best configuration is SVM with SMOTE, achieving macro F1 of 0.752 and accuracy of 0.784. Class weighting benefits Random Forest (+0.011) but slightly reduces SVM, confirming that linear SVM on TF-IDF is robust to moderate imbalance at IR = 2.41. Per-issuer evaluation reveals macro F1 variation from 0.647 on TPIA to 0.881 on BBNI, shaped by vocabulary consistency, class dominance, and domain specificity. These results provide a transparent and reproducible classical baseline that situates transformer-based and deep-learning approaches on ID-SMSA within a well-defined reference frame.