Extreme class imbalance remains a persistent challenge in machine learning, particularly in high-impact domains such as fraud detection, medical diagnosis, and risk analysis, where minority classes represent critical outcomes. Conventional models often fail in such settings due to their bias toward majority classes, resulting in poor minority detection despite high overall accuracy. Although various data-level and algorithm-level techniques have been proposed, existing studies typically evaluate them in isolation and lack a comprehensive understanding of their effectiveness across different imbalance conditions. To address this gap, this study proposes a systematic comparative framework that integrates data-level resampling, cost-sensitive learning, and hybrid approaches to evaluate their performance under varying imbalance ratios and noise levels. Multiple benchmark datasets are utilized, and experiments are conducted using standardized preprocessing, controlled imbalance simulation, and repeated trials to ensure robustness. Performance is assessed using imbalance-aware metrics, including precision, recall, F1-score, and ROC-AUC. The results indicate that hybrid approaches consistently outperform standalone methods, achieving the most stable and balanced performance across all scenarios. In particular, hybrid models demonstrate superior minority class recall and F1-score while maintaining competitive precision, especially under extreme imbalance conditions. The primary goal of this research is to provide a comprehensive evaluation of imbalance-handling strategies and offer practical guidance for selecting appropriate techniques based on dataset characteristics. The findings highlight the importance of combining data-centric and model-centric approaches to enhance robustness and reliability in imbalanced learning environments. The results demonstrate up to a 32% improvement in recall compared to baseline models.
Tamsir Ariyadi, E. Noche, Nisha Pandey et al.· Journal of Data Science· 0 citations
The widespread deployment of machine learning (ML) systems in critical domains has exposed the limitations of accuracy-centric evaluation, particularly under conditions involving class imbalance, noise, and distributional shifts. Existing studies frequently employ alternative metrics in isolation and lack a unified framework capable of systematically assessing model robustness, reliability, and decision sensitivity across varying data conditions. To address this gap, this study proposes a structured multi-metric evaluation framework that integrates classification, ranking-based, calibration, and robustness-oriented metrics for comprehensive ML performance assessment. A quantitative experimental design is employed using multiple benchmark datasets with varying statistical characteristics, including balanced and imbalanced distributions. Controlled perturbation scenarios—including noise injection, class imbalance manipulation, and distribution shift simulation—are introduced to emulate realistic deployment environments. Several machine learning models, namely Logistic Regression, Support Vector Machines, Random Forest, and Multi-Layer Perceptron (MLP), are evaluated using metrics such as Accuracy, F1-score, ROC-AUC, PR-AUC, Brier Score, and Expected Calibration Error (ECE). The experimental results demonstrate that accuracy consistently overestimates model effectiveness under adverse conditions, while alternative metrics reveal substantial hidden weaknesses in minority class detection and probability reliability. Among the evaluated models, MLP achieved the strongest overall performance, obtaining a ROC-AUC of 0.94 and PR-AUC of 0.89 under baseline conditions. Furthermore, calibration-oriented metrics exhibited significantly higher sensitivity to perturbation severity compared to accuracy. This study contributes to the advancement of trustworthy artificial intelligence by promoting a comprehensive, context-aware, and robustness-oriented evaluation framework capable of supporting more reliable real-world ML deployment.
Ade Putra, E. Noche, Diksha D. Gabhane· Journal of Data Science· 0 citations
Machine learning models are increasingly deployed in decision-critical environments such as healthcare, finance, and autonomous systems. However, most conventional models generate deterministic predictions without quantifying uncertainty, which can lead to overconfident mispredictions when data are noisy, incomplete, or outside the training distribution. This limitation exposes a critical gap between predictive accuracy and decision reliability in real-world Al systems. To address this challenge, this study proposes an uncertainty-aware machine learning framework that integrates probabilistic modeling techniques into conventional predictive architectures to jointly estimate epistemic and aleatoric uncertainty. The proposed framework enables models to produce predictive distributions rather than single point predictions, allowing systems to quantify confidence and identify high-risk predictions. Experiments were conducted on multiple benchmark datasets representing both classification and regression tasks under varying levels of noise and data incompleteness. The experimental results demonstrate that the proposed framework achieves predictive performance comparable to deterministic baselines while significantly improving reliability and uncertainty calibration. In classification tasks, the model maintained competitive accuracy and F1-scores while providing well-calibrated confidence estimates, whereas in regression experiments the approach reduced prediction risk by identifying high-error cases through increased uncertainty variance. Robustness tests further show that the framework effectively signals degraded prediction reliability when encountering noisy or incomplete inputs. These findings indicate that incorporating uncertainty estimation enhances trustworthiness and robustness without sacrificing predictive performance. The study highlights uncertainty modeling as a critical component for developing reliable and responsible Al systems capable of supporting risk-sensitive decision-making in real-world data-driven environments
Evi Yulianingsih, E. Noche, V. Yadav et al.· Journal of Data Science· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
