Bridging the Modality Gap With Differentiable Intensity Rendering for Online LiDAR-Camera Calibration

Abstract

Precise spatial alignment between cameras and LiDAR is a prerequisite for robust multi-modal perception, yet this alignment is often disrupted by mechanical vibrations or drifts during operation. To recalibrate sensors on the fly using naturally collected scene data, 3D Gaussian Splatting-based methods have emerged as promising differentiable solutions. However, current frameworks generally focus on RGB and depth consistency, and have not fully leveraged the reflectance information provided by LiDAR intensity, which limits their ability to fully bridge the modality gap. Bridging this gap, we propose a novel online calibration framework that incorporates LiDAR intensity as a learnable Gaussian attribute, enabling the generation of dense, differentiable intensity maps from sparse LiDAR point clouds. To effectively align these rendered intensity maps with camera grayscale images, we adopt a Modality Independent Neighborhood Descriptor loss that captures local self-similarity patterns rather than absolute pixel values. Experiments on public driving benchmarks validate that leveraging differentiable LiDAR intensity improves online LiDAR–camera calibration.