Unboxing the Black Box: A Survey on Mechanistic Interpretability for Algorithmic Understanding of Neural Networks
TL;DR
It is argued that mechanistic interpretability has the potential to support a more scientific understanding of machine learning systems – treating models not only as tools for solving tasks, but also as systems to be studied and understood.
Abstract
The black box nature of deep neural networks poses a significant challenge for the deployment of transparent and trustworthy artificial intelligence (AI) systems. With the growing presence of AI in society, it becomes increasingly important to develop methods that can explain and interpret the decisions made by these systems. To address this, mechanistic interpretability (MI) emerged as a promising and distinctive research program within the broader field of explainable artificial intelligence (XAI). MI studies the inner computations of neural networks and translates them into human-understandable algorithms. It encompasses reverse-engineering techniques aimed at uncovering the computational algorithms implemented by neural networks. In this paper, we present a comprehensive survey of mechanistic interpretability, synthesizing a rapidly growing and fragmented body of work into a single, structured reference. We propose a unified taxonomy of MI approaches and provide a detailed analysis of key techniques, illustrated with concrete examples and pseudo-code. We contextualize MI within the broader interpretability landscape, comparing its goals, methods, and insights to other strands of XAI. Additionally, we trace the development of MI as a research area, highlighting its conceptual roots and the accelerating pace of recent work. We argue that MI has the potential to support a more scientific understanding of machine learning systems – treating models not only as tools for solving tasks, but also as systems to be studied and understood. We intend this survey to serve as an entry point for new researchers to the field of mechanistic interpretability.