Skip to content
Open access

Deployment-Oriented Evaluation of LLM-Generated Optimization Code: Repairability, Compatibility, and Constraint-Coverage in Parallel-Machine Scheduling

2026 · IEEE Access · Vol 14, pp. 138842-138864 · 0 citations · 22 references
Computer Science

TL;DR

GenSE-Scheduler, a deployment-oriented framework for evaluating LLM-generated parallel-machine scheduling code, compares monolithic schedulers with four-stage modular pipelines under typed contracts, sandboxed execution, a failure taxonomy, stage-swap repair, and pre-composition compatibility prediction.

Abstract

Large language models (LLMs) increasingly generate executable optimization code, yet evaluations often rank programs by objective value, overlooking deployment-relevant properties such as validity under structural constraint changes, failure localization, repairability, and component compatibility. We introduce GenSE-Scheduler, a deployment-oriented framework for evaluating LLM-generated parallel-machine scheduling code. It compares monolithic schedulers with four-stage modular pipelines under typed contracts, sandboxed execution, a failure taxonomy, stage-swap repair, and pre-composition compatibility prediction. We evaluate 1,350 artifacts from four locally served open-weight model families across 1,039,600 artifact–instance pairs. Modular pipelines exhibit substantially greater outcome-level behavioral diversity than monolithic artifacts (Cliff’s <inline-formula> <tex-math notation="LaTeX">$\delta = +0.799$ </tex-math></inline-formula> for cluster entropy; <inline-formula> <tex-math notation="LaTeX">$\delta = +1.000$ </tex-math></inline-formula> for pairwise regret distance). However, modular generation does not meet the pre-registered regret-equivalence criterion (<inline-formula> <tex-math notation="LaTeX">$\Delta = +0.565 \gt \varepsilon = 0.05$ </tex-math></inline-formula>), although the effect is negligible (<inline-formula> <tex-math notation="LaTeX">$\delta = +0.044$ </tex-math></inline-formula>) among passing in-distribution pairs. Among failing modular pipelines, 97 of 98 (99.0%) admit at least one valid single-stage substitution, but this repairability is contract-level: the resulting swaps rarely meet tight scheduling-quality tolerances. Repairability is also strongly stage-dependent, with the <monospace>assign</monospace> stage as the main bottleneck. A static Module Compatibility Predictor reaches macro-F<inline-formula> <tex-math notation="LaTeX">$1=0.872$ </tex-math></inline-formula> for predicting compatible swaps before composition. Machine-eligibility stress tests—using a constraint that is present in the generation prompts but never instantiated in the C1–C4 calibration regime—yield a 0% pass rate in both an isolated eligibility class and a combined release-date/setup-time/eligibility hold-out, which we interpret as a joint calibration-coverage and serialization-contract boundary, not evidence that LLMs cannot implement eligibility constraints when those constraints are explicitly instantiated and exercised. Against a classical greedy min-completion-time list-scheduling baseline, neither architecture is makespan-competitive: the heuristic attains far lower regret and is matched or beaten by well under 1% of passing artifact–instance pairs. Overall, modular generation does not dominate monolithic generation in raw scheduling quality; rather, modular decomposition makes deployment-relevant properties measurable.

Read PDF

Similar papers

Open access Oct 2026

CAST: A Compiler-Based Framework for Systematically Testing LLM Compositional Safety

This paper proposes CAST, a systematic testing framework designed to evaluate the compositional safety of LLMs in the domain of malicious code, and demonstrates that CAST systematically exposes severe safety violations in strongly aligned models that resist conventional red-teaming.

Lu Yan, Zhuo Zhang, Xiang-Zhe Xu et al. · 0 citations

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

It is argued that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and this is supported with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies.

Om Nepal, Sushant Aryal, Oluseyi Olukola et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SOVER: Formal Certification of Optimization Reformulations via LLM-Assisted SMT Verification

SOVER, an LLM-assisted SMT framework that separates semantic mapping from formal certification, is introduced, and Z3 checks domain cross-feasibility and global objective-order preservation for mixed-integer linear formulations, while dReal provides tolerance-aware feasibility/range and $\epsilon$-argmin checks for con...

Swapnil Bhattacharyya, Mayank Baranwal · 0 citations
#natural language process... Preprint Sep 2026

$\Phi$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

This work presents $\Phi$-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization.

Lei-Lei Ding, Shu-Min Wang, Yu-Ting Huang et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.