Results show that LLMs can complement compiler analysis as speculative semantic proposers, provided that their artifacts are validated and evaluated.
Abstract
Optimizing compilers miss profitable transformations when their enabling semantics are absent from the analyzed program representation. We ask whether large language models (LLMs) can recover such semantics from heterogeneous C/C++ context and realize them as validated, contract-preserving artifacts. We introduce SeGaBench, an executable benchmark containing 100 synthetic and 20 source-backed cases spanning low-level assumptions, data-structure invariants, and high-level semantic lifting. Each case includes hidden enabling semantics, an oracle artifact, correctness and semantic validators, and a reproducible performance protocol. We evaluate five LLMs using five independent responses per case. The strongest model produces correct artifacts in 94.8% of responses, achieves at least 1.05x speedup in 83.3%, and obtains a performance success on 93.3% of cases. Nevertheless, correct artifacts often close only part of the oracle gap. These results show that LLMs can complement compiler analysis as speculative semantic proposers, provided that their artifacts are validated and evaluated.
Large Language Models (LLMs) have transformed software engineering by automating code generation, yet they frequently produce code that is syntactically correct but behaviorally incorrect—failing to respect repository conventions, misapplying practices across contexts, or violating semantic properties. These failures come from a fundamental limitation: current LLMs learn syntactic patterns but lack behavioral understanding of how code executes in context. This work addresses three interconnected challenges in building behaviorally-grounded code generation: (1) retrieving not just what symbols are, but how they behave through usage patterns; (2) adapting practices across contexts by separating behavioral intent from implementation details; and (3) enforcing semantic properties during generation without prohibitive runtime costs. Preliminary results demonstrate substantial improvements: 48.2% exact match on repository completion (vs. 29.05% baseline) and 100% conformance on cross-framework practice transfer (vs. 0% baseline). This work aims to transform LLMs from pattern-matching systems into behaviorally-aware generation system that understand what code does, not just what it looks like.
Large language models have performed impressively in code generation tasks, yet it remains unclear whether they understand code semantics and whether this affects their ability to write high-quality code. To address this question, we introduce SemBench, a novel benchmark consisting of 1000 diverse C programs sourced from the CodeParrot GitHub-code dataset, with 15,404 semantic questions spanning six basic but fundamental properties: dead code-statement, data dependency, function reachability, dominator, dead code-loop, and liveness. We evaluate 16 widely-used models across 7 families. Even the best model reaches only 80.42% accuracy, and failure rates range from 19.58% to 86.01%, showing a substantial gap between code generation and static semantic understanding. Performance varies sharply across semantic categories. Function reachability shows the strongest correlation with HumanEval and MBPP among the tested categories, suggesting that specific static semantic abilities may partly explain code-generation success. Overall, our experiments underscore the substantial gap between the static semantic understanding and code completion capabilities of modern LLMs.
Jade Xu, Renliang Sun, Zijian Ding et al.· Communications AI & Computin...· 0 citations
Large Language Models (LLMs) are now widely used for code generation, yet even syntactically correct output may contain logical and semantic errors that remain invisible until runtime, particularly in framework-driven applications, where correctness depends on dependency injection, framework conventions, configuration, library compatibility, and database interaction. This paper presents GenTest, an open-source platform for context-aware dynamic generation of Java code using LLMs, runtime compilation and class loading, Spring bean registration, JPA/PostgreSQL execution, and declarative YAML-based test validation. GenTest supports multiple LLM providers through a provider-agnostic interface and evaluates generated components within a live Spring Boot application context. Across 209 test cases and 933 assertions, GenTest achieves a 63.9% assertion pass rate, and 94.2% of failures occur after successful compilation, confirming that execution-centered evaluation reveals failure modes invisible to static analysis.
Muhammed Suphi Şeyhkuruş, Tolga Ovatman· Annual International Compute...· 0 citations
Differential compiler testing requires automatically generated programs that are not only diverse and bug-revealing, but also semantically well-defined and reproducible. Rule-based generators provide strong validity guarantees but offer limited control over semantic variation, while large language models (LLMs) can synthesize expressive programs without principled mechanisms for balancing competing testing objectives. This paper proposes LMOEC, a constrained multi-objective evolutionary framework that integrates code language models as semantic genetic operators within an NSGA-II search process. Instead of using the LLM as a one-shot generator, we employ it for population initialization, crossover, and mutation at the program level, enabling semantics-aware recombination while preserving strict admissibility constraints. Compiler test generation is formulated as a multi-objective optimization problem that simultaneously promotes structural diversity, cross-configuration output inconsistency, semantic complexity, and robustness to mutation. A constraint-driven acceptance pipeline enforces syntactic validity, deterministic execution, bounded runtime, and avoidance of undefined behavior before evolutionary selection. By maintaining a Pareto front of non-dominated programs, LMOEC preserves multiple high-value test archetypes reflecting different trade-offs between bug exposure and reproducibility. The framework demonstrates how expressive code models can be systematically embedded into evolutionary multi-objective optimization for reliability-critical software testing.
Lang Hong Nguyet Anh, Ho Viet Duc Luong, Vu Van An· Annual Conference on Genetic...· 0 citations
Large language models (LLMs) are increasingly applied to reverse-engineering tasks, and recent threat-intelligence reporting shows them operating inside live offensive-security workflows. Claims about their capability, however, outpace our ability to measure it. Existing benchmarks for LLM-assisted binary analysis treat the construction of function-level ground truth as a solved pre-processing step and report accuracy without disclosing how many functions were reliably evaluable. We argue that the principal obstacle to fair evaluation is not model capability but the reliability of binary-to-source alignment under compiler optimization. This paper presents Reforge, a provenance-tracked pipeline that constructs function-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, and that operationalizes alignment uncertainty as an eight-gate confidence funnel with three-tier stratification. On a controlled micro-benchmark, high-confidence yield falls from 87.2% to 65.9% across optimization levels, and unpaired comparisons overstate optimization-induced performance decay through survivorship bias. A proof-of-concept evaluation of seven contemporary LLMs on function naming demonstrates the validity of the concept and generally motivates an uncertainty-aware benchmarking practice.
It is shown that generative compilation reduces non-compiling outputs and improves functional correctness, relative to standard post-generation feedback, by detecting a broad range of errors close to their source and early during generation, thereby reducing errors cascades and enabling focused diagnostics.
Niels Mündler-Sasahara, Hristo Venev, Dawn Song et al.· 0 citations