From Vibe to Spec: An Empirical Study to Demystify Specifications in Code Generation

Abstract

Recent advances in large language models (LLMs) for code generation have given rise to vibe coding, a programming practice in which developers interact with LLMs through unstructured natural language and primarily validate implementations by inspecting overall execution outputs. However, the inherent ambiguity and inconsistency of natural language present substantial challenges for generating complex and large-scale software systems. To mitigate these limitations, an emerging AI programming paradigm—specification-driven development—advocates providing LLMs with precise and consistent specifications to guide code generation. Despite its promise, fundamental questions regarding the appropriate forms of specifications and the granularity at which they should be supplied to LLMs remain largely unexplored. To address this gap, we present the first empirical study that systematically investigates the impact of two orthogonal dimensions of specifications—form and granularity—on the effectiveness of LLM-based code generation. We construct a dataset comprising five medium-sized software projects, ranging from several hundred to over one thousand lines of code, implemented in multiple programming languages and spanning diverse application domains (e.g., system utilities and games). For each project, we design specifications in three forms—natural language, coarse-grained pseudocode, and fine-grained pseudocode—to capture the form dimension. To examine granularity, we further decompose the fine-grained pseudocode into four levels of input granularity: line-level, function-level, module-level, and project-level. Using this dataset, we evaluate code generated by three representative models, GPT-5, DeepSeek-V3.1, and Qwen-3.7. Evaluation is conducted using a comprehensive set of metrics, including text-similarity-based metrics (e.g., BLEU-4 and CodeBLEU), test-based metrics (e.g., accuracy), and human-centered metrics (e.g., usability and logical consistency). Our results demonstrate that using fine-grained, structured pseudocode specifications significantly enhances generation stability and quality. Regarding granularity, while coarser-grained specifications (project and module level) generally achieve higher code quality, organizing specifications at the module level provides the best overall balance between quality, stability, and token efficiency. Moreover, additional analyses indicate that this module-level granularity yields the most user-friendly outcome for human developers.