Looks Good, but Does It Run? A Large-Scale Empirical Study of the Executability of Example Code in Model Cards

Abstract

Model cards document how machine learning models should be understood, evaluated, and reused in practice. Many also provide example code to help users access and use models, yet the executability of such code remains understudied. We take Hugging Face as an open model repository and conduct an empirical study of model-card example code. Among 291 model cards, 68.38% of the example-code snippets fail to run, suggesting that execution failures are common and may hinder model reuse. To characterize these failures, we construct a taxonomy comprising fifteen error categories and twenty-four subcategories and distill fifty reusable repair strategies. Based on these findings, we develop CardCodeFixer, a tool for automated repair. Experimental results show that CardCodeFixer increases executability from 29.63% to 56.08% with GPT-4o and from 31.75% to 51.85% with DeepSeek-v3. These results characterize execution problems in model-card example code and show that automated repair can improve executability.