It is found that MultiMAE is a surprisingly strong out-of-the-box baseline, transferring effectively even to modalities it has never seen, while targeted modifications yield incremental but consistent improvements on multi-sensor transfer and overall robustness.
Abstract
Transfer learning from pretrained backbones has become standard practice in multi-sensor computer vision---yet two questions remain open: which pretrained weights transfer best to multi-sensor setups, and can in-domain pretraining outperform off-the-shelf solutions? Frameworks like MultiMAE demonstrate the utility of masked autoencoding in a multi-modal context, but rely entirely on dense, model-derived modalities. It is unclear how robustly these methods translate to real-world applications that use physical sensors, where the aggregation of diverse public datasets for large-scale pretraining inherently introduces modality sparsity. To investigate this systematically, we curate a multi-sensor autonomous driving collection of $2.82M$ images from 16 publicly available datasets, incorporating both physical hardware modalities (RGB, LWIR, NIR, Event) and model-derived modalities (Stereo Depth, Segmentation). We use this collection for in-domain pretraining based on the MultiMAE approach. We adapt MultiMAE's loss formulation to accommodate sparse samples, and study the effect of an enforced encoder bottleneck for modality alignment and of additional losses specific to dense reconstruction (SSIM and FFT). We then systematically compare against common pretrained backbones (ViT, MAE, DINO and MultiMAE) across downstream sensor fusion tasks. We find that MultiMAE is a surprisingly strong out-of-the-box baseline, transferring effectively even to modalities it has never seen, while our targeted modifications yield incremental but consistent improvements on multi-sensor transfer and overall robustness.
RL-Axial, a multimodal segmentation framework with interpretable, sample-adaptive context aggregation, is proposed, which trains a lightweight selector with a group-normalized policy-gradient objective based on per-sample relative rewards.
Dong Xing, Hang Yang, Jin-He Zhang et al.· Cognitive Computation· 0 citations
Human Activity Recognition (HAR) on resource-constrained edge devices relies on lightweight models, but robustly training these architectures is severely bottlenecked by the scarcity of diverse, annotated Inertial Measurement Unit (IMU) data. To overcome this empirical limitation, we propose a cross-modal Knowledge Dis...
M. E. P. Monteiro, Jamil de Araujo Farhat, B. Nassu et al.· IEEE Access· 0 citations
Multimodal perception is central to many computer vision applications, including robotics and autonomous driving, where different sensing modalities provide complementary cues. Self-supervised learning offers a scalable solution for exploiting large amounts of unlabeled multimodal data. However, existing methods either...
Carlo Sgaravatti, Daniele Kopyshevskiy, Davide Secco et al.· 0 citations
Feed-forward 3D reconstruction models have achieved impressive performance by scaling model and dataset size, but their cost excludes most research groups and precludes edge deployment. Additionally, generating 3D supervision without sensors still relies on slow, unreliable Structure-from-Motion, as the community lacks...
Vision foundation models have achieved remarkable success in monocular depth estimation, yet they tightly couple semantic and structural cues within a single RGB observation. This coupled representation makes them fragile to real-world imaging degradations such as motion blur and exposure failure, and inherently incomp...
Xiang Chen, Ya-Peng Meng, Yu-Guo Chen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.