Comparing the use of supervised machine learning variable selection methods in the context of two-group classification in the psychological and health sciences

Abstract

Introduction Variable selection (VS) is crucial for building accurate and generalizable classification models. Reducing the necessary number of variables improves model efficiency, interpretability, and generalizability while reducing data collection burden. Despite the availability of various VS methods, their comparative performance within machine learning frameworks for classification remains unclear. Methods This study conducted a large-scale Monte Carlo simulation to compare VS methods from distinct families: regularization techniques (LASSO, Elastic Net), tree-based methods (random forest-based Boruta wrapper), penalized support vector machines, and metaheuristics (genetic algorithm). An empirical example from an alcohol use disorder dataset illustrated the comparative findings. Supplementary simulations evaluated four additional filter methods, re-estimated performance under stratified 10-fold cross-validation, and quantified overfitting via train-test performance gaps. Results In the main simulation, the genetic algorithm demonstrated the strongest overall classification accuracy, while LASSO offered the most parsimonious solution with minimal performance loss. All three top-performing methods (GA, LASSO, Elastic Net) substantially reduced the number of variables from the saturated model. Supplementary analyses aligned with main simulation findings. Discussion These findings highlight the practical value of machine learning VS methods in psychological research by demonstrating how to balance accuracy, interpretability, and scalability. The results provide guidance for researchers selecting among competing VS approaches based on their priorities regarding model performance versus parsimony.