Autonomous Road Infrastructure Monitoring via Multi-modal Data Fusion using Non-dedicated Vehicle Sensors

Abstract

Highway pavement networks require frequent condition assessment, but conventional inspections with dedicated survey vehicles remain costly and are conducted at long intervals. This delays timely maintenance and increases life-cycle costs. This study presents an AI-based pavement condition assessment framework based on multi-modal sensor fusion, integrating video and vibration data from a vehicle-mounted smartphone with distributed fiber-optic sensing (FOS) installed along the roadside. These two data collection approaches are complementary, as the smartphone sensors cover the full network at low cost but produce noisier data, whereas FOS offers high-fidelity measurements only at equipped sections. Data from all three modalities were collected over multiple runs on a Korean expressway and evaluated across seven scenarios covering both individual and fused sensor configurations. Three tree-based classifiers (LightGBM, XGBoost, Random Forest) were trained to predict the International Roughness Index (IRI), a roughness-based index derived from the road's longitudinal profile, and the Highway Pavement Condition Index (HPCI), a composite index that incorporates both roughness and surface distress. The results indicate that vibration features contributed effectively to both IRI and HPCI by reflecting the vehicle's response to surface irregularities. Vision features contributed primarily to HPCI by capturing surface defects, while FOS features improved both predictions by providing structural response measurements inaccessible to the vehicle-mounted sensors. The findings demonstrate that combining modalities generally outperformed individual modalities, but the most effective combination differed by target index, offering practical guidance on how to pair sensing modalities for different pavement condition metrics.