When Iterative Prompting Fails: An Empirical Study of Unit Test Generation with Open-Source LLMs

Abstract

Large language models (LLMs) have shown promise in automated unit test generation, yet the effectiveness of prompt engineering for small, locally-deployed open-source models remains poorly understood. Following growing interest in local LLM deployment to mitigate data exposure risks, this paper presents a controlled empirical study using Mistral 7B, Phi-3 Mini 4K Instruct, and CodeLlama 7B Instruct across four prompting strategies and a prompt-only generate-critique-refine pipeline.We first find that increasing prompt structure raises syntactic validity from 0.15 (zero-shot) to 0.58 (few-shot), yet semantic correctness does not follow: pass rate peaks at just 0.055 under instruction-based prompting and collapses to 0.000 under few-shot prompting (p > 0:05, Wilcoxon signed-rank test). We then show that iterative refinement further degrades performance across all models. Pass rate on Mistral 7B drops monotonically from 0.061 at T0 to 0.030 at T2; CodeLlama 7B deteriorates sharply from 0.549 to 0.189; and Phi-3 shows only a transient gain at T1 (0.171, up from 0.128 at T0) before reverting to near-baseline at T2 (0.122).Failure analysis identifies three recurring failure modes: error propagation, semantic drift, and corruption of previously correct outputs (Success → SyntaxError). These results demonstrate that prompt-only iterative refinement is unreliable for semantic correctness in unit test generation, and that stronger external feedback mechanisms—such as execution-guided refinement or verifier-based filtering—are necessary for meaningful progress.