A large-scale empirical study analyzing performance evolution in 15 mature open-source Java projects hosted on GitHub, showing that different project domains and scales exhibit distinct performance characteristics, which suggest the need for domain-specific performance management strategies that account for the project scales.
An automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok is presented and a substantial gap between correctness and quality attributes is revealed.
Seyed Mohammad Mahdi Ghalandarian, Majid Bazargani, Masoumeh Taromirad· 0 citations
MCR-Bench is introduced, the first defect state-aware benchmark designed for realistic multi-round code review, and in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.
De-Wu Zheng, Yan-Lin Wang, Xi-Wen Wang et al.· Proceedings of the ACM on So...· 1 citation
Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, an...
Om Nepal, Sushant Aryal, Oluseyi Olukola et al.· 0 citations
An empirical study involving 5 Large Language Models and 4 benchmarks evaluates the effectiveness and efficiency of 3 widely used adequacy criteria: statement coverage, branch coverage, and mutation testing, finding that mutation testing only marginally outperforms traditional coverage criteria in both triggering and d...
Asma Hamidi, Michael Konstantinou, R. Degiovanni et al.· 0 citations
This research investigates Large Language Models (LLMs), which are considered among the most performant AI models to date, for the CVD task, and high-light the importance of tuning to resolve the task, the performance of Double tuning, as well as the potential of Llama models for CVD.
This paper presents a formal mathematical model for categorizing the outcome of generated-tests into four classes, a couple of basic metrics: Bug-Revealing Rate (BRR) and Bug-Validating Rate (BVR); and two basic statistical tests to ensure that the results are rigorous.
Zeyad Farooq Lutfi· Al-Noor Journal of Engineeri...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
