Prompts Matter: On the Impact of Code Smell Remediation with Large Language Models

Abstract

Code smells accumulate as technical debt, reducing software maintainability. Refactoring is the established remediation practice, but it is complex and often deprioritized. Large Language Models can support automated refactoring, but their effectiveness depends on how they are prompted. In this study, we evaluate 14 prompt configurations with GPT-OSS and Qwen3 on 2,000 Java and Python functions, producing 168,000 function-level refactorings. We evaluate behavior preservation, code smell remediation, and cost by token proxy. Results show that prompt effectiveness is contextual. For GPT-OSS, prompts combining explicit smell thresholds with reasoning triggers increased remediation odds by up to 1.32× over zero-shot at a 1.05–1.10× cost multiplier. For Qwen3, prompts eliciting reasoning reduced behavior-preservation odds to 0.75× of zero-shot and raised cost by up to 1.51× zero-shot. Conversely, the threshold-only prompt increased remediation odds by up to 1.18× with near-neutral overhead (1.01–1.02×). Cost per successful refactoring was 1.01–1.04× ZS for GPT-OSS and 1.12–1.21× for Qwen3. Overall, prompt choice matters, but model choice produced the larger difference. These findings support validation-centered, model-specific prompt selection rather than a uniform strategy.