An Empirical Study of Classical and Deep Learning Models for Software Defect Prediction

Abstract

Software defect prediction (SDP) is a critical endeavor for enshancing software reliability and optimizing quality assurance efforts by enabling the early identification of fault-prone components within the development lifecycle. This study presents a comprehensive empirical evaluation comparing the efficacy of classical machine learning (ML) techniques and deep learning (DL) models for SDP. We utilize four widely recognized NASA benchmark datasets (KC1, JM1, PC1, MC1), each characterized by static code metrics. Following rigorous data preprocessing, which included normalization of feature scales and resolution of label inconsistencies, eight classical ML models, Random Forest, Gradient Boosting, AdaBoost, Support Vector Machine, Logistic Regression, k-Nearest Neighbors, Naïve Bayes, and Decision Tree, were trained and assessed across multiple performance measures. To explore the capabilities of deep learning, a multilayer perceptron (ANN) served as a baseline, complemented by a Long Short-Term Memory (LSTM) network. Our findings reveal that ensemble-based ML models consistently deliver superior performance across all datasets, including those with significant class imbalance, such as MC1, where detecting the minority class is particularly challenging. The ANN model also demonstrates competitive performance on balanced datasets. A key contribution of this research lies in elucidating the nuanced potential of the LSTM model. While its performance is susceptible to severe class imbalance, LSTM exhibits notable advantages in moderately imbalanced datasets (e.g., PC1), demonstrating higher recall for the minority class and improved F1-scores. This suggests that deep learning architectures, despite their capacity to capture richer feature interactions, remain sensitive to data imbalance and diversity issues. Consequently, the study reinforces the robustness of conventional ML methods for static metric-based defect prediction while simultaneously highlighting specific conditions under which deep learning, particularly LSTM, can offer distinct advantages by identifying deeper structural relationships within the metric space. This work contributes to the ongoing discourse on effective SDP strategies by providing a detailed comparative analysis and identifying the strengths and limitations of both classical ML and DL approaches in varied data contexts. The insights gained are valuable for researchers and practitioners aiming to select appropriate models for specific SDP challenges, particularly concerning class imbalance.