An enhanced framework for automated vehicle crash severity prediction under unobserved heterogeneity using natural language processing and data augmentation

Abstract

Objective Autonomous vehicle (AV) crashes continue to occur, indicating the need for a comprehensive understanding of their characteristics to support safety management in future mixed-traffic environments. Previous studies employing text mining or rule extraction have faced limitations in generalizing findings due to small, heterogeneous datasets and severe class imbalance in crash severity levels. To address these gaps, this article develops an integrated framework that combines clustering, latent important topic of crash pattern, data augmentation to improve severe crash prediction using AV crash reports.Method This article develops a multi-stage prediction framework that integrates established methods, including Bayesian Latent Class Analysis (LCA) to capture unobserved heterogeneity, Latent Dirichlet Allocation (LDA) to derive thematic factors from unstructured crash narratives, and a Wasserstein Generative Adversarial Network with Gradient Penalty (WGAN-GP) for minority-class data augmentation, to AV crash data. The analysis used AV crash reports from the California Department of Motor Vehicles, excluding nonmoving cases. LCA identified three distinct latent subclasses characterized by unique contexts, road environments, and vulnerable road user (VRU) interactions. Topic modeling revealed class-specific narrative patterns, enriching structured datasets with semantic variables. Three classifiers were evaluated under different sampling methods.Results CatBoost combined with WGAN-GP and topic-enhanced features achieved the highest F1-score (0.730), outperforming both SMOTE-based oversampling and original imbalanced datasets. The results confirm that integrating semantic information from narratives with deep generative oversampling significantly improves severe crash prediction performance.Conclusions Although the article is based on historical traffic crash data, this article supports proactive AV safety assessment by transforming the crash narratives and structured variables into influencing insights on severe traffic crash risk. The integrated application of various modeling identifies latent crash patterns, that are often obscured in imbalanced AV crash datasets. These insights can help policymakers and manufacturers prioritize scenario-based testing, monitor recurring high-risk conditions involving road environments and design treatments for mixed-traffic environments. Rather than claiming methodological novelty or broad adaptability, this study demonstrates the practical value of combining established methods to analyze heterogeneous and imbalanced AV crash data.