CloudIR-Bench is introduced, the first publicly available benchmark providing a dataset and evaluation framework for incident report-based RCA and mitigation prediction, and results reveal the capabilities and limitations of existing LLMs in this domain.
Abstract
Root cause analysis (RCA) and mitigation prediction are essential for incident management, enabling the identification of underlying causes of failures and ensuring the reliability of complex cloud-based systems. With rising incident volume and system complexity, incident reports have become a valuable resource for these tasks. Recent studies have shown promising results using Large Language Models (LLMs) to predict root causes and mitigation actions from textual descriptions in incident reports. Despite these advances, further progress is hindered by the lack of publicly available datasets; existing work relies on private data, which limits transparency, reproducibility, and generalization. To address this gap, we introduce CloudIR-Bench, the first publicly available benchmark providing a dataset and evaluation framework for incident report-based RCA and mitigation prediction. Curated from incident reports across five leading cloud-based systems, CloudIR-Bench captures a diverse range of real-world production scenarios representing authentic failures and engineers'reasoning. We also release an open-source evaluation framework that includes eight recent LLM-based approaches. Our comprehensive evaluation includes zero-shot and in-context learning experiments, along with a qualitative analysis to identify where and why LLMs fail. Benchmarking results reveal the capabilities and limitations of existing LLMs in this domain. The benchmark, including the dataset, framework and supplementary materials, is available at https://github.com/hoangtnm/CloudIR-Bench.
Hero is presented, a hypothesis-centered RCA framework that formulates diagnosis as controlled discrimination among competing hypotheses via bounded hypothesis-space instantiation, explicit belief-state tracking, and ambiguity-guided evidence acquisition to suggest that explicitly controlling competing hypotheses is an...
Jiewei Lyu, Jun-Quan Yi, Shu-Fu Liang et al.· Proceedings of the 41st IEEE...· 0 citations
We present OpenStack-RCA-Bench, an open dataset of 64 chaos-engineering incidents for benchmarking RCA methods in OpenStack IaaS. It features a 9-phase pipeline integrating four fault injection methods (service stop, port block, process kill, config corruption), automated log collection via Loki, and ground truth defin...
I. Bogomolov, O. Borisenko· Proceedings of the 41st IEEE...· 0 citations
As autonomous agents advance, the traditional boundary between software development and cloud operations gradually blurs, increasing the need for integrated incident detection, localization, and remediation. Existing benchmarks evaluate isolated diagnostic queries or repository-level program repair, failing to test con...
Enterprise cloud migration is commonly justified by expectations of improved availability and operational resilience. Many organizations find that cloud transition does not automatically deliver those outcomes, even after adopting multi-zone deployments, managed services, and cloud-native recovery patterns. This paper...
Karthik Jambulingam· 2026 IEEE International Conf...· 0 citations
The results show that telemetry-grounded investigation is practical for small teams, and that self-reported confidence must not replace human review, and that self-reported confidence must not replace human review.
J. Veerendeswari, V. Karunagaran, S. Sujeethan et al.· Indian Journal of Computer S...· 0 citations
Synthetic event data generation has emerged as a promising means of supporting process mining and Business Process Management (BPM) when real event logs are unavailable, sensitive, or unsuitable for controlled experimentation. Existing approaches range from traditional process-model simulation and rule-based generation...
M. Kuhn, Yorck Zisgen, Joscha Grüger et al.· Process Science· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.