This work presents LoopCoder pre-trained on 12T+ code and general tokens, along with LoopCoder-Thinking and LoopCoder-Instruct variants, the first large-scale looped transformer for code, achieving comparable performance to standard dense architectures with more parameters.
Looped Mamba and Looped Hybrid Mamba-Transformer architectures, which repeatedly apply a shared Mamba block to introduce explicit finite-depth recurrent computation, are investigated and adaptive exit-state selection improves downstream performance at intermediate depths, while actual inference-time savings require additional state-handling mechanisms.
Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo et al.· 0 citations
Large Language Models (LLMs) are predominantly assessed based on their common sense reasoning, language comprehension, and logical reasoning abilities. While models trained in specialized domains like mathematics or coding have demonstrated remarkable advancements in logical reasoning, there remains a significant gap in evaluating their code generation capabilities. Existing benchmark datasets fall short in pinpointing specific strengths and weaknesses, impeding targeted enhancements in models’ reasoning abilities to synthesize code.
To bridge this gap, this thesis introduces two novel contributions: CodeEval and CodeQual. CodeEval is an innovative, pedagogical benchmarking method that mirrors the evaluation processes encountered in academic programming courses. It comprises a multi-dimensional benchmark dataset of 602 hand-crafted problems designed to rigorously evaluate LLMs across 24 distinct aspects of Python programming, covering three proficiency levels—beginner, intermediate, and advanced—and includes both class-based and function-based problem types with detailed problem specifications and comprehensive test suites achieving 99.1% coverage. To facilitate widespread adoption, we developed RunCodeEval, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline. Our evaluation of 15 state-of-the-art LLMs revealed consistent performance degradation with increasing complexity (validated statistically, Cohen’s d = 0.790) and universal struggles with advanced concepts like concurrency.
Code quality is inherently subjective, encompassing dimensions like readability, efficiency, and adherence to language idioms that traditional static metrics fail to capture adequately. While large language models can assess these subjective qualities, lightweight models offer practical advantages: seamless CI/CD pipeline integration, lower operational costs, and full control over model behavior. We investigate whether such models can learn to assess code quality by training on synthetic LLM annotations. We introduce CodeQual, a dataset of 5,819 code samples derived from five established sources spanning diverse domains—competitive programming, pedagogical problems, software engineering, and general benchmarks—scored by LLMs across five quality dimensions, with 655 human-annotated samples for evaluation. Our fine-tuned model, CodeQualBERT, not only matches LLM performance but exceeds inter-human agreement on all five dimensions, achieving 16–100% improvement over the inter-human agreement baseline.
Together, these contributions provide a comprehensive framework for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluation.
This work evaluates native and retrofitted looped language models on API-Bank, BFCL, and NESTful, comparing looped and non-looped models trained under matched supervised fine-tuning recipes and varying recurrent depth at inference time to suggest that looped language models are a promising architecture for agentic systems that require reliable planning, coordination, and execution of compositional tool use workflows.
A. C. Popescu, Haitz Sáez de Ocáriz Borde, Pietro Liò· 0 citations
Deep learning (DL) compilers such as Apache TVM translate high-level models into optimized low-level code through multi-stage compilation pipelines. While recent testing efforts have improved fuzzing of optimization stages, they still face two key challenges: (i) the lack of semantics-preserving test models, leading to low validity, and (ii) coarse-grained input generation that fails to trigger hard-to-reach compiler components. To address these limitations, we propose CovCraft, a unified testing framework that integrates constraint-driven model generation with large language model (LLM)-guided input adaptation. CovCraft constructs diverse and valid ONNX models via symbolic constraint encoding and SMT solving, and then iteratively refines inputs using LLM-guided prompts to target uncovered functions, enabling the activation of rarely executed code paths. We evaluate CovCraft on TVM and observe consistent improvements over state-of-the-art techniques: it increases branch and function coverage by 8.9% and 7.0%, respectively, and detects 8 bugs. Moreover, the LLM-guided component achieves an 83.75% success rate in covering designated target functions, demonstrating the effectiveness of combining constraint-based generation with adaptive LLM reasoning for DL compiler testing. The prototype implementation of CovCraft is publicly available at: https://github.com/duduhedangdang/CovCraft.
Yifei He, Fangyu Yang, Ting Su et al.· Annual International Compute...· 0 citations
Large Language Models (LLMs) have become the dominant workload on modern AI accelerators, yet deploying them on specialized hardware still faces two core challenges: how to import a trained model into a compiler-friendly intermediate representation, and how to efficiently schedule the autoregressive inference loop under limited on-chip memory. This paper presents an MLIR (Multi-Level Intermediate Representation) based compilation method for large language models, illustrated using two dialects of operators, TopOp and TpuOp. TopOp serves as a high-level graph dialect that is independent of both the source framework and the target chip, and is responsible for expressing model semantics; TpuOp serves as the target hardware dialect, carrying chip-related decisions such as quantization, layer groups, and memory layout. A model is first represented as TopOp, then lowered layer by layer to TpuOp, and finally a deployable binary is generated. In addition, each Transformer layer is split into three stages for static compilation: prefill, prefill_kv (prefill with historical key-value cache), and decode, so as to accommodate the different computational characteristics of prompt-parallel processing and per-token generation. The method has been implemented in the TPU-MLIR compiler {https://github.com/sophgo/tpu-mlir} and the LLM-TPU deployment project {https://github.com/sophgo/LLM-TPU}, supporting a variety of generative models including the Qwen, Llama, InternVL, and MiniCPM-V series, as well as multiple quantization and deployment forms such as GPTQ, AWQ, and AutoRound.
Pengchao Hu, Zhibin Xin, Yifan Chen et al.· 0 citations
This work proposes a novel Reinforcement Learning from AI Feedback (RLAIF) framework, finding that specifications can be iteratively refined by leveraging feedback from the LLM's own outputs, creating a self-improving loop.
Jaykithan Y Patel· SIGSOFT FSE Companion· 0 citations