Revisiting TuRTLe: A Comprehensive Evaluation of LLMs for RTL Generation

Abstract

Rapid advancements in LLMs have driven the adoption of generative AI in domains like Electronic Design Automation (EDA). Within the field of software development, EDA presents unique challenges derived from specific requirements of generated RTL code; RTL code must not only be syntactically correct and functionally accurate, but also synthesizable by hardware generators, while matching performance, power and area (PPA) constraints. These additional requirements introduce complexities that existing code-generation benchmarks often fail to capture, limiting their effectiveness in evaluating LLMs for RTL generation. To address this gap, we propose TuRTLe, a unified evaluation framework designed to systematically assess LLMs across key RTL generation tasks. TuRTLe integrates multiple existing benchmarks and automates the evaluation process, enabling a comprehensive assessment of LLM performance in syntax correctness, functional correctness, synthesis, PPA optimization, and exact line completion. Using this framework, a diverse set of forty open LLMs are assesed, tracking their strengths and weaknesses in EDA-specific tasks. Our results identify the best match for specific tasks (e.g., base models are better in module completion tasks, instruct-tuned models are better in specification-to-RTL tasks), while finding that recent models with autoregressive reasoning chain perform the best overall. We also analyze common compiler and runtime failures, study correlations between benchmarks and evaluation goals, and investigate potential training-data contamination in existing RTL datasets. These analyses provide further insight into the capabilities and limitations of current benchmarks for RTL generation.