TSGen: Automated Troubleshooting Guide Generation
TL;DR
TSGen, an automated pipeline for generating high-quality, structured TSGs from historical incident reports using large language models (LLMs), consists of filtering and classifying incident data into diagnostically relevant categories and distilling core incidents to ensure diversity and generalizability.
Abstract
Maintaining up-to-date troubleshooting guides (TSGs) is critical for the reliability of cloud systems, yet manual maintenance often leads to inefficiencies and outdated documentation. This paper proposes TSGen, an automated pipeline for generating high-quality, structured TSGs from historical incident reports using large language models (LLMs). Our approach consists of three stages: (1) filtering and classifying incident data into diagnostically relevant categories, (2) distilling core incidents to ensure diversity and generalizability, and (3) organizing the distilled knowledge into a directed acyclic graph (DAG) that captures root causes and resolutions in a structured manner. By leveraging real-world incident discussions, TSGen produces dynamic and reusable guides tailored for live troubleshooting. Experiments on real-world incidents from Microsoft demonstrate that TSGen achieves 54.8% incident coverage and approximately 3× higher retrieval accuracy compared to baselines. Furthermore, the system supports iterative updates, allowing guides to evolve alongside dynamic cloud environments. Human evaluation shows that on-call engineers rate these generated TSGs significantly higher than human-crafted ones.