Comparative Analysis of Model‐Agnostic Explanation Methods in Materials Science

Abstract

Machine learning models have become a powerful tool in materials science, accelerating the discovery process by accurately predicting molecular properties. However, the black‐box nature of many of these models makes it difficult to understand or validate their predictions. Consequently, explainable artificial intelligence (XAI) has emerged to make predictive models more transparent. This need for explainability is crucial in chemistry and materials science, as identified structure–property relationships can guide the design of novel molecules. However, there is still a lack of XAI benchmarks for materials science. To address this gap, this study presents a comparative quantitative and qualitative analysis of six selected XAI methods on molecular fingerprints, commonly used representations for material property prediction tasks. Moreover, this work explores the use of mean and robust rank aggregation (RRA) to combine multiple XAI approaches. The results reveal significant discrepancies in the feature importance rankings generated by different XAI methods, demonstrating that the choice of explainer can introduce bias and alter scientific interpretation in the material discovery process. Specifically, while general‐purpose tabular explainers like SHAP proved highly effective, chemistry‐specific counterfactuals often failed to correctly capture important features. Given the methods’ disagreements, aggregating the explanations, especially using RRA, can yield a well‐balanced compromise.