AI-generated C++ code has a distinct quality profile, showing higher rates of interface and coupling burdens, copy and allocation overheads, and a reliance on explicit loops over optimized standard APIs, which translates into tangible downstream costs, including increased review effort and a 5-8% increase in compute resource consumption.
Abstract
The widespread integration of AI coding assistants offers undeniable boosts to engineering velocity. Yet, recent studies point to a growing trade-off, revealing persistent challenges with code quality and maintainability. Industry leaders, including frontier AI labs, echo these concerns. As large language models are increasingly relied upon to author production code, understanding their impact on shipped software quality has become a critical priority. However, assessing these effects in industrial workflows remains difficult due to observability barriers. We study the impact of AI-generated code on production quality within a large enterprise operating global products relied upon by billions of users daily. Driven by this scale and user trust, the organization values code quality and has built thorough observability for every line of code deployed into production, enabling us to overcome measurement barriers to assess these effects. This study presents a large-scale empirical analysis of AI-generated C++ code from April 2025 to April 2026, tracking 3.52 million code changes across this enterprise's brownfield codebase. The core purpose is to understand the quality, performance, and maintenance characteristics of AI-generated code compared to human-written code in a production environment at scale. We find that AI-generated C++ code has a distinct quality profile, showing higher rates of interface and coupling burdens, copy and allocation overheads, and a reliance on explicit loops over optimized standard APIs. These issues translate into tangible downstream costs, including increased review effort and a 5-8% increase in compute resource consumption. However, we demonstrate that providing models with targeted, taxonomy-informed feedback can mitigate these effects, leading to an 11.1% reduction in targeted static analysis warnings and improved computational efficiency.
The use of AI agents for automatic code generation has become increasingly common in software development. However, concerns remain about the quality of the generated code, including aspects of maintainability, readability, and long-term evolution. This study compares the structural quality of code produced by three widely adopted vibe coding tools --- Lovable, v0, and Replit --- starting from a single generation prompt. We generate three independent projects per tool, totalling nine web applications, and submit them to static analysis with SonarQube. We collect metrics such as the number of issues, severity distribution, estimated remediation effort, cyclomatic and cognitive complexity, and code duplication. Preliminary results show that the tools exhibit distinct qualitative profiles: Lovable concentrates issues of lower severity but presents a substantially higher density of code smells per KLOC, while v0 and Replit produce more code with more aggressive severity profiles. These findings suggest that choosing between vibe coding tools involves structural trade-offs that go beyond perceived productivity.
Large Language Models (LLMs) are increasingly used to generate production code, yet systematic methods for evaluating their quality and security remain underdeveloped. This tutorial introduces a reusable, end-to-end evaluation pipeline grounded in empirical software engineering practices, focusing on post-generation validation rather than prompt design. Participants will apply static analysis tools to assess maintainability, reliability, and security, and compare results across models, prompts, and human-written baselines. The pipeline supports structured aggregation and interpretation of outputs, enabling reproducible and defensible assessments. Extensions include agentic remediation, explainability for trust calibration, and bias-aware evaluation. Attendees will leave with practical evaluation artifacts and a principled framework for validating AI-generated code in modern development workflows.
The software engineering landscape is undergoing a radical transition from manual syntax craftsmanship to "Vibe Coding"a paradigm defined by prompt-driven, intent-based software generation. This research quantifies the impact of this shift on the software development lifecycle (SDLC), specifically evaluating the efficacy of code review and debugging. Utilizing a diagnostic pipeline with a 50-developer cohort, this study compares traditional Integrated Development Environment (IDE) workflows against AI-assisted "vibe" workflows (utilizing GitHub Copilot and ChatGPT). Our findings demonstrate a profound "Time Displacement Penalty." While initial development velocity approaches zero, severe debugging penalties and elevated bug escape rates in complex systems fundamentally offset these early gains. We identify a "Cognitive Void" wherein developers exhibit high speed and syntax correctness but suffer from a measurable detachment from system architecture and forensic comprehension. This detachment facilitates "Automation Bias" during peer reviews, where the aesthetic perfection of AI-generated syntax masks deep, hallucinated logic failures. The study concludes that while Vibe Coding is optimal for isolated prototyping, it introduces systemic risks in mission-critical architecture. The data mandates a pivot in software engineering pedagogy from syntax authorship to rigorous forensic curation and adversarial auditing.
Prateek Sharma Kharel, Suman Thapalia· International Journal For Mu...· 0 citations
An automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok is presented and a substantial gap between correctness and quality attributes is revealed.
Seyed Mohammad Mahdi Ghalandarian, Majid Bazargani, Masoumeh Taromirad· 0 citations
: The growing adoption of large language models (LLMs) in software engineering has introduced new opportunities but also risks in the software maintenance lifecycle. While LLMs can generate entire codebases from natural language prompts, such automatically generated or rapidly prototyped code often accumulates structural debt, making systematic refactoring increasingly urgent. This work investigates LLMs as metric-driven refactoring assistants rather than code generators. Six models (ChatGPT, Claude, Gemini, Grok, DeepSeek, and Qwen) were evaluated on two types of Java projects: three controlled applications with manually inflated structural metrics, and three real-world applications from public GitHub repositories. Using MetricsReloaded in IntelliJ IDEA, we measured four CK metrics: complexity (WMC), cohesion (LCOM), coupling (CBO), and inheritance depth (DIT). Results indicate that LLMs significantly reduce complexity and coupling, improving class simplicity and modularity. However, cohesion improvements remained limited, with LCOM proving especially elusive. Inheritance depth showed strong reductions in synthetic high-metric applications but minimal change in real projects. ChatGPT produced the most consistent and structurally stable refactoring outputs in real applications, though occasional cohesion deterioration occurred. These findings suggest that while LLMs are valuable assistants for structural improvement, their interventions require careful monitoring to avoid unintended trade-offs.
Tindwende Sawadogo, Fadel Touré· Proceedings of the 21st Inte...· 0 citations
The recent meteoric rise of LLMs (Large Language Models) and associated tools was largely unexpected and surprising to most. The rapid ascent of this technology has caught many software developers unawares, leaving them suddenly somewhat ignorant, and arguably under-skilled.
LLMs, whilst still advancing, have recently demonstrated impressive capabilities in their ability to assist software developers in their day-to-day tasks (e.g., coding new features, and locating and fixing issues). However, the use and adoption of LLMs presents many larger challenges for society as a whole; many of which are not in themselves technical concerns.
This paper examines the current and perceived impact of this technology in the context of Open Source. We identify several social, economic, environmental, political, legal, and technical concerns regarding the use of LLMs in Open Source projects.
We contribute guidance around defining an AI Policy for Open Source projects. We further offer an AI Policy Score Card to assist projects in clearly defining and declaring how they wish to work with AI or not.
Adam Retter· Balisage Series on Markup Te...· 0 citations