Generating Behavior-Driven Development Artifacts

Abstract

Behavior-Driven Development (BDD) scenarios coexist with large volumes of semi-structured records (e.g., documentation, issues, informal feature descriptions), and keeping the two synchronized is laborintensive. We present a study of bidirectional generation 11https://github.com/Artin-Biniek/Submission-Project between records and BDD scenarios, comparing finetuning of CodeT5+ encoder–decoder models and the Qwen2.5-Coder-0.5B decoder-only model against retrievalaugmented generation (RAG) with few-shot prompting on Meta-Llama-3.1-8B, CodeT5+, and DeepSeek-Coder-33B. On a curated dataset of 2,100 aligned pairs, finetuning attains best BLEU/F1 of 0.9394/0.9549 and Exact Match of 0.8119 for record generation, while RAG never exceeds BLEU 0.35 and yields Exact Match 0.00. Twoannotator human evaluation, two LLM judges, and twoproportion Z-tests confirm that fine-tuning significantly outperforms RAG $(p<0.001)$ for high-fidelity BDD generation.