Deployment-Oriented Evaluation of LLM-Generated Optimization Code: Repairability, Compatibility, and Constraint-Coverage in Parallel-Machine Scheduling
GenSE-Scheduler, a deployment-oriented framework for evaluating LLM-generated parallel-machine scheduling code, compares monolithic schedulers with four-stage modular pipelines under typed contracts, sandboxed execution, a failure taxonomy, stage-swap repair, and pre-composition compatibility prediction.
Abstract
Large language models (LLMs) increasingly generate executable optimization code, yet evaluations often rank programs by objective value, overlooking deployment-relevant properties such as validity under structural constraint changes, failure localization, repairability, and component compatibility. We introduce GenSE-Scheduler, a deployment-oriented framework for evaluating LLM-generated parallel-machine scheduling code. It compares monolithic schedulers with four-stage modular pipelines under typed contracts, sandboxed execution, a failure taxonomy, stage-swap repair, and pre-composition compatibility prediction. We evaluate 1,350 artifacts from four locally served open-weight model families across 1,039,600 artifact–instance pairs. Modular pipelines exhibit substantially greater outcome-level behavioral diversity than monolithic artifacts (Cliff’s <inline-formula> <tex-math notation="LaTeX">$\delta = +0.799$ </tex-math></inline-formula> for cluster entropy; <inline-formula> <tex-math notation="LaTeX">$\delta = +1.000$ </tex-math></inline-formula> for pairwise regret distance). However, modular generation does not meet the pre-registered regret-equivalence criterion (<inline-formula> <tex-math notation="LaTeX">$\Delta = +0.565 \gt \varepsilon = 0.05$ </tex-math></inline-formula>), although the effect is negligible (<inline-formula> <tex-math notation="LaTeX">$\delta = +0.044$ </tex-math></inline-formula>) among passing in-distribution pairs. Among failing modular pipelines, 97 of 98 (99.0%) admit at least one valid single-stage substitution, but this repairability is contract-level: the resulting swaps rarely meet tight scheduling-quality tolerances. Repairability is also strongly stage-dependent, with the <monospace>assign</monospace> stage as the main bottleneck. A static Module Compatibility Predictor reaches macro-F<inline-formula> <tex-math notation="LaTeX">$1=0.872$ </tex-math></inline-formula> for predicting compatible swaps before composition. Machine-eligibility stress tests—using a constraint that is present in the generation prompts but never instantiated in the C1–C4 calibration regime—yield a 0% pass rate in both an isolated eligibility class and a combined release-date/setup-time/eligibility hold-out, which we interpret as a joint calibration-coverage and serialization-contract boundary, not evidence that LLMs cannot implement eligibility constraints when those constraints are explicitly instantiated and exercised. Against a classical greedy min-completion-time list-scheduling baseline, neither architecture is makespan-competitive: the heuristic attains far lower regret and is matched or beaten by well under 1% of passing artifact–instance pairs. Overall, modular generation does not dominate monolithic generation in raw scheduling quality; rather, modular decomposition makes deployment-relevant properties measurable.
This paper proposes CAST, a systematic testing framework designed to evaluate the compositional safety of LLMs in the domain of malicious code, and demonstrates that CAST systematically exposes severe safety violations in strongly aligned models that resist conventional red-teaming.
Lu Yan, Zhuo Zhang, Xiang-Zhe Xu et al.· Proceedings of the ACM on So...· 0 citations
It is argued that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and this is supported with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies.
Om Nepal, Sushant Aryal, Oluseyi Olukola et al.· arXiv.org· 0 citations
SOVER, an LLM-assisted SMT framework that separates semantic mapping from formal certification, is introduced, and Z3 checks domain cross-feasibility and global objective-order preservation for mixed-integer linear formulations, while dReal provides tolerance-aware feasibility/range and $\epsilon$-argmin checks for con...
A platform-agnostic CPU baseline is established by evaluating five open-weight models below two billion parameters and revealing a reliability-resource trade-off for edge deployment.
This work presents $\Phi$-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization.
Lei-Lei Ding, Shu-Min Wang, Yu-Ting Huang et al.· 1 citation