BCaLLM: Call Graph-Guided Python Breaking Change Detection with Large Language Models

Abstract

Software libraries frequently evolve, introducing breaking changes that disrupt client applications. Existing detection approaches primarily target static programming languages or focus on syntactic changes, leaving behavioral breaking changes in dynamic languages such as Python underexplored. This task is particularly challenging due to side effects and call relationships, two critical factors that implicitly alter API behaviors and propagate change impact across library APIs. To address these challenges, we propose a generalized taxonomy of function API breaking changes. Grounded in Hyrum’s Law, our taxonomy is defined from the client’s perspective of observable behaviors and unifies both syntactic and behavioral categories in a multi-label formulation. Furthermore, we present BCaLLM, a novel framework to detect fine-grained breaking changes in Python packages by leveraging call graphs and large language models (LLMs). BCaLLM constructs a fused call graph to scope change impact, prunes compatible APIs and code context via memory-based heuristics, and employs an LLM to detect specific breaking changes. We construct PyBCEval, a manually annotated benchmark of 588 APIs from 27 version pairs of 19 widely used Python packages. Experiments with diverse LLMs show that BCaLLM outperforms text-based baselines by 3.71%–10.16% and LLM-based baselines by 1.60%–4.83% in F1-score.