Eagle, a comprehensive benchmarking framework tailored for evaluating OpsLLMs, delivers a deployable foundation for advancing large-model applications in AIOps and open-source the framework and dataset to foster community adoption and reproducibility.
Abstract
The rapid growth in scale and complexity of modern software systems has intensified the need for intelligent and reliable IT operations. While Artificial Intelligence for IT Operations (AIOps) addresses some challenges, existing solutions predominantly rely on isolated, task-specific models that struggle with interpreting multimodal data, incur high maintenance costs, and lack sufficient transparency. Operations Large Language Models (OpsLLMs) offer unified, knowledge-rich reasoning capabilities, yet their evaluation faces significant barriers, including the absence of Ops-centric evaluation taxonomies, limited availability of public datasets, simplistic question-generation methods, and inadequate quality standards for comprehensive operations tasks. We present Eagle, a comprehensive benchmarking framework tailored for evaluating OpsLLMs. Deployed inside Huawei, Eagle ingests enterprise product documentation and synthesizes 4,845 domain-grounded QA pairs across logs, metrics, traces, and configurations, which are paired with a standardized model evaluation system. This deployment supported multiple evaluations of OpsLLM and culminated in an internal horizontal benchmarking report that informed model selection and rollout decisions. Methodologically, Eagle (i) defines an operations-centric taxonomy aligning core LLM abilities with end-to-end operations tasks; (ii) implements an automated question-generation pipeline with multi-granular quality controls validated by human annotation; and (iii) provides reproducible evaluation suites and metrics for scenario-driven reasoning. In offline studies, Eagle-generated test suites improve expert-rated rubric scores by 22%–49% over state-of-the-art baselines, enabling more precise assessments of anomaly detection, fault diagnosis, and root-cause analysis abilities in OpsLLMs. To foster community adoption and reproducibility, we open-source the framework1 and a sanitized dataset2. By bridging general LLM evaluation and operations practice, Eagle delivers a deployable foundation for advancing large-model applications in AIOps.
While Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, their effectiveness in unit testing is often constrained by insufficient context regarding external dependencies. This limitation is particularly pronounced in industrial settings, where proprietary code remains opaque to the model. To address this challenge, we present a systematic empirical study of multiple strategies for context enrichment and optimization in LLM‐based unit test generation, conducted on seven diverse projects (three open‐source and four proprietary industrial systems), encompassing 261 distinct methods. By evaluating seven implementations (ranging from basic prompts to optimized context reduction strategies) across 10 independent runs, we analysed a total of 28,710 test suites. Our results demonstrate that combining prompt engineering with external dependency retrieval achieves an average branch coverage increase of 11.52 percentage points on industrial software over the baseline, with statistically significant improvements across all competing implementations. Beyond coverage, richer context substantially reduces generation‐repair iterations, cutting median execution time by 51.3% in industrial projects. We further show that reducing external dependencies to method signatures alone decreases input token consumption by up to 46.6% (25.4% in industrial projects) while fully preserving the coverage and efficiency gains of the complete retrieval approach. To confirm that these benefits are not tied to a specific model, we replicate the core comparison across three LLM backends from different families, obtaining a consistent, statistically significant coverage improvement on industrial code in every case. These findings establish this optimized context strategy as a cost‐effective solution for scalable, industrial‐grade automated test generation.
Javier Ferrer, Francisco Chicano· Expert systems· 0 citations
The majority of data in businesses and industries is stored in tables, databases, and data warehouses. Reasoning with table-structured data poses significant challenges for large language models (LLMs) due to its hidden semantics, inherent complexity, and structured nature. One of these challenges is lacking an effective evaluation benchmark fairly reflecting the performances of LLMs on broad table reasoning abilities. In this paper, we fill in this gap by presenting a comprehensive table reasoning benchmark, TReB. Firstly, we propose a taxonomy to systematically measure both shallow table understanding abilities and deep table reasoning abilities, covering a total of 26 sub-tasks. We then construct a high quality dataset through a dedicated data processing and synthesis procedure. Based on these well-constructed samples, we design an evaluation framework to robustly measure table reasoning capabilities with three distinct inference modes. Experimental results with our data and framework reveal that existing LLMs still have significant room for improvement in addressing the complex and real world table related tasks. Both the dataset and evaluation framework are publicly available, with the dataset hosted on huggingface.co/datasets/JT-LM/JIUTIAN-TReB, and the framework on github.com/JT-LM/jiutian-treb.
Ce Li, Xiaofan Liu, Zhiyan Song et al.· Annual International ACM SIG...· 3 citations
AutoDavis is introduced, a first-of-its-kind automatic and dynamic evaluation protocol that enables on-demand benchmarking of LVLMs across specific capability dimensions and shows effectiveness and reliability, offering a new paradigm for dynamic benchmarking of multimodal intelligence.
Han Bao, Yue Huang, Yanbo Wang et al.· Proceedings of the 32nd ACM...· 0 citations
Large Language Models (LLMs) are predominantly assessed based on their common sense reasoning, language comprehension, and logical reasoning abilities. While models trained in specialized domains like mathematics or coding have demonstrated remarkable advancements in logical reasoning, there remains a significant gap in evaluating their code generation capabilities. Existing benchmark datasets fall short in pinpointing specific strengths and weaknesses, impeding targeted enhancements in models’ reasoning abilities to synthesize code.
To bridge this gap, this thesis introduces two novel contributions: CodeEval and CodeQual. CodeEval is an innovative, pedagogical benchmarking method that mirrors the evaluation processes encountered in academic programming courses. It comprises a multi-dimensional benchmark dataset of 602 hand-crafted problems designed to rigorously evaluate LLMs across 24 distinct aspects of Python programming, covering three proficiency levels—beginner, intermediate, and advanced—and includes both class-based and function-based problem types with detailed problem specifications and comprehensive test suites achieving 99.1% coverage. To facilitate widespread adoption, we developed RunCodeEval, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline. Our evaluation of 15 state-of-the-art LLMs revealed consistent performance degradation with increasing complexity (validated statistically, Cohen’s d = 0.790) and universal struggles with advanced concepts like concurrency.
Code quality is inherently subjective, encompassing dimensions like readability, efficiency, and adherence to language idioms that traditional static metrics fail to capture adequately. While large language models can assess these subjective qualities, lightweight models offer practical advantages: seamless CI/CD pipeline integration, lower operational costs, and full control over model behavior. We investigate whether such models can learn to assess code quality by training on synthetic LLM annotations. We introduce CodeQual, a dataset of 5,819 code samples derived from five established sources spanning diverse domains—competitive programming, pedagogical problems, software engineering, and general benchmarks—scored by LLMs across five quality dimensions, with 655 human-annotated samples for evaluation. Our fine-tuned model, CodeQualBERT, not only matches LLM performance but exceeds inter-human agreement on all five dimensions, achieving 16–100% improvement over the inter-human agreement baseline.
Together, these contributions provide a comprehensive framework for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluation.
A large-scale benchmark with 60\sim 90× more queries than prior work, built on 3× more databases, an automated pipeline that can execute existing methods without manual intervention, and multi-dimensional, fine-grained evaluation metrics for comprehensive assessment.
Bo Li, Chenzhan Wang, Longkang Lin et al.· Proceedings of the 32nd ACM...· 0 citations
The increasing deployment of large language models (LLMs) in enterprise environments creates reliability challenges related to hallucination, factual inconsistency, limited knowledge traceability, uncertainty, and operational efficiency. This study develops a literature-based architectural framework for reliable knowledge retrieval systems that separates external knowledge management from LLM-based reasoning and generation. The framework synthesizes architectural mechanisms for knowledge representation, hybrid retrieval, reranking, evidence selection, context construction, response verification, provenance tracking, uncertainty handling, guardrails, and computational efficiency. The resulting architecture organizes these mechanisms into coordinated layers that control the flow of external evidence from knowledge sources to generated responses while supporting traceability and controlled abstention when sufficient evidence is unavailable. The architectural synthesis further identifies complementary strategies for enterprise deployment, including semantic caching, model routing, and human oversight, to balance reliability, scalability, and operational cost. The analysis indicates that reliable LLM deployment should be treated as an end-to-end architectural problem rather than solely a model-performance problem, with knowledge access, evidence quality, verification, provenance, and governance functioning as integrated system components. The proposed framework provides a structured foundation for designing maintainable, auditable, and reliable knowledge retrieval systems for enterprise and other high-stakes applications
Bharat Kumar Reddy Karumuri· International Journal of Eng...· 0 citations