Evaluating the Robustness of Counterfactual Generating Methods for Tree-Based Ensembles

Abstract

Counterfactuals (CFs) are a widely studied approach for explaining machine learning models. They aim to identify a set of minimal and meaningful perturbations in input features required to alter a prediction of a model, thereby offering insights into decision boundaries. For instance, a CF may describe the changes needed for a rejected loan application to be accepted. Despite their usefulness, CFs are sensitive to model dynamics. Updates to training data, modifications of model parameters, or shifts in the underlying distribution can alter decision surfaces, potentially invalidating previously generated explanations. This instability raises critical concerns about the robustness and reliability of CFs in practical applications. Although tree-based ensemble models built using bagging and boosting are extensively used in real-world contexts, understanding the robustness of CFs for these models is still in its infancy. To address this gap, this work conducts an empirical investigation into the robustness of CF explanations for tree-based classifiers. In this work, several publicly available CF generation methods are evaluated across a range of experimental settings involving both bagging and boosting tree-based classifier models. Moreover, a novel robustness-oriented CF generation method is proposed that exploits the ensemble of trees in the classifier models. The results reveal substantial variation in robustness across methods and model types, raising concerns about the reliability of CFs in dynamic environments. This study advances understanding of the limitations and potential of existing techniques, and provides practical guidance for practitioners applying CFs to tree-based ensemble models. The code used for our experiments is available on https://github.com/VinuraD/CFRobustness-of-Trees.