A Robust Heterogeneous Ensemble Framework for Software Cost Estimation with Prediction Uncertainty Quantification

Abstract

Successfully estimating software costs is critical to project management because poor estimates lead to overruns, scheduling issues, and project failure. Machine learning improves estimates, but usually only point estimates are produced without providing valid uncertainty around them. This paper presents a method using an uncertainty-aware heterogeneous ensemble of 100 bootstrap-trained base learners (using gradient boosting, extra trees, and random forests), which jointly produce point predictions and prediction intervals using a robust trimmed mean, with effort modeled on the log scale. Using the NASA93 dataset (with 93 projects, split 80/20 into 74 for the training set and 19 for the test set) and repeated cross-validation (feature selection and scaling were performed only within the training folds to avoid leakage), the model achieved a mean absolute error of 309.1 and a percentage of relative error deviation from the predicted value rate of 51.3% (30 out of 74 projects) outperforming all eight of the Bayesian baselines it was compared to, while being the only model to provide the validity of its prediction intervals through calibration. Given that normality of the ensemble predictions is rejected, empirical and distribution-free percentiles (with 89.5% of the prediction intervals containing the test project true outcomes) are used because they are not biased by the training data. The average interval width covered by the validation of those prediction intervals to cross-validate the interval width over 86.1% of the test set differs significantly from the linear bases using a paired Wilcoxon test (P < 0.001). Thus, the framework described couples competitive levels of accuracy with quantifiable and empirically validated confidence levels.