Skip to content
Preprint

CloudIR-Bench: A Benchmark for Root Cause Analysis and Mitigation of Cloud Incidents

Oct 2026 · 0 citations · 46 references
Computer Science

TL;DR

CloudIR-Bench is introduced, the first publicly available benchmark providing a dataset and evaluation framework for incident report-based RCA and mitigation prediction, and results reveal the capabilities and limitations of existing LLMs in this domain.

Abstract

Root cause analysis (RCA) and mitigation prediction are essential for incident management, enabling the identification of underlying causes of failures and ensuring the reliability of complex cloud-based systems. With rising incident volume and system complexity, incident reports have become a valuable resource for these tasks. Recent studies have shown promising results using Large Language Models (LLMs) to predict root causes and mitigation actions from textual descriptions in incident reports. Despite these advances, further progress is hindered by the lack of publicly available datasets; existing work relies on private data, which limits transparency, reproducibility, and generalization. To address this gap, we introduce CloudIR-Bench, the first publicly available benchmark providing a dataset and evaluation framework for incident report-based RCA and mitigation prediction. Curated from incident reports across five leading cloud-based systems, CloudIR-Bench captures a diverse range of real-world production scenarios representing authentic failures and engineers'reasoning. We also release an open-source evaluation framework that includes eight recent LLM-based approaches. Our comprehensive evaluation includes zero-shot and in-context learning experiments, along with a qualitative analysis to identify where and why LLMs fail. Benchmarking results reveal the capabilities and limitations of existing LLMs in this domain. The benchmark, including the dataset, framework and supplementary materials, is available at https://github.com/hoangtnm/CloudIR-Bench.

View source

Similar papers

Book Open access Oct 2026

HERO: Hypothesis-Centered Root-Cause Analysis for Microservice Incidents

Hero is presented, a hypothesis-centered RCA framework that formulates diagnosis as controlled discrimination among competing hypotheses via bounded hypothesis-space instantiation, explicit belief-state tracking, and ambiguity-guided evidence acquisition to suggest that explicitly controlling competing hypotheses is an...

Jiewei Lyu, Jun-Quan Yi, Shu-Fu Liang et al. · 0 citations
Book Open access Oct 2026

OpenStack-RCA-Bench: A Reproducible IaaS Root Cause Analysis Dataset

We present OpenStack-RCA-Bench, an open dataset of 64 chaos-engineering incidents for benchmarking RCA methods in OpenStack IaaS. It features a 9-phase pipeline integrating four fault injection methods (service stop, port block, process kill, config corruption), automated log collection via Loki, and ground truth defin...

I. Bogomolov, O. Borisenko · 0 citations
Preprint Nov 2026

SWEOps-Bench: Evaluating Agents for End-to-End Resolution of Software-Induced Cloud Incidents

As autonomous agents advance, the traditional boundary between software development and cloud operations gradually blurs, increasing the need for integrated incident detection, localization, and remediation. Existing benchmarks evaluate isolated diagnostic queries or repository-level program repair, failing to test con...

Dan-Wen Chen, Guang-Ba Yu, Peng-Fei Chen · 0 citations
Conference Open access Sep 2026

The Cloud Resilience Assumption Gap: A Taxonomy of Cloud-Native Failure Modes and Practitioner Validation

Enterprise cloud migration is commonly justified by expectations of improved availability and operational resilience. Many organizations find that cloud transition does not automatically deliver those outcomes, even after adopting multi-zone deployments, managed services, and cloud-native recovery patterns. This paper...

Karthik Jambulingam · 0 citations
Review Open access Oct 2026

Agentic AI Incident Response Platform for Cloud Applications: Telemetry-Grounded Root-Cause Investigation with Human Approval

The results show that telemetry-grounded investigation is practical for small teams, and that self-reported confidence must not replace human review, and that self-reported confidence must not replace human review.

J. Veerendeswari, V. Karunagaran, S. Sujeethan et al. · 0 citations
Review Open access Oct 2026

A systematic review of synthetic event data generation: overview, challenges, and research directions

Synthetic event data generation has emerged as a promising means of supporting process mining and Business Process Management (BPM) when real event logs are unavailable, sensitive, or unsuitable for controlled experimentation. Existing approaches range from traditional process-model simulation and rule-based generation...

M. Kuhn, Yorck Zisgen, Joscha Grüger et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.