Skip to content
Preprint

Do Pretrained Backbones Transfer to Physical Sensors? A Systematic Study of Multimodal Initialization Across the Sensor Stack

Nov 2026 · 0 citations · 43 references
Computer Science

TL;DR

It is found that MultiMAE is a surprisingly strong out-of-the-box baseline, transferring effectively even to modalities it has never seen, while targeted modifications yield incremental but consistent improvements on multi-sensor transfer and overall robustness.

Abstract

Transfer learning from pretrained backbones has become standard practice in multi-sensor computer vision---yet two questions remain open: which pretrained weights transfer best to multi-sensor setups, and can in-domain pretraining outperform off-the-shelf solutions? Frameworks like MultiMAE demonstrate the utility of masked autoencoding in a multi-modal context, but rely entirely on dense, model-derived modalities. It is unclear how robustly these methods translate to real-world applications that use physical sensors, where the aggregation of diverse public datasets for large-scale pretraining inherently introduces modality sparsity. To investigate this systematically, we curate a multi-sensor autonomous driving collection of $2.82M$ images from 16 publicly available datasets, incorporating both physical hardware modalities (RGB, LWIR, NIR, Event) and model-derived modalities (Stereo Depth, Segmentation). We use this collection for in-domain pretraining based on the MultiMAE approach. We adapt MultiMAE's loss formulation to accommodate sparse samples, and study the effect of an enforced encoder bottleneck for modality alignment and of additional losses specific to dense reconstruction (SSIM and FFT). We then systematically compare against common pretrained backbones (ViT, MAE, DINO and MultiMAE) across downstream sensor fusion tasks. We find that MultiMAE is a surprisingly strong out-of-the-box baseline, transferring effectively even to modalities it has never seen, while our targeted modifications yield incremental but consistent improvements on multi-sensor transfer and overall robustness.

View source

Similar papers

Open access Oct 2026

RL-Axial: Learning When to Compute Global Context for Multimodal Segmentation

RL-Axial, a multimodal segmentation framework with interpretable, sample-adaptive context aggregation, is proposed, which trains a lightweight selector with a group-normalized policy-gradient objective based on per-sample relative rewards.

Dong Xing, Hang Yang, Jin-He Zhang et al. · 0 citations
Open access 2026

Cross-Modal Knowledge Distillation From Synthetic Video for IMU-Based HAR

Human Activity Recognition (HAR) on resource-constrained edge devices relies on lightweight models, but robustly training these architectures is severely bottlenecked by the scarcity of diverse, annotated Inertial Measurement Unit (IMU) data. To overcome this empirical limitation, we propose a cross-modal Knowledge Dis...

M. E. P. Monteiro, Jamil de Araujo Farhat, B. Nassu et al. · 0 citations
#artificial intelligence Preprint Nov 2026

DenseTwins: Dense Modality Decoupling for Multimodal Self-Supervision

Multimodal perception is central to many computer vision applications, including robotics and autonomous driving, where different sensing modalities provide complementary cues. Self-supervised learning offers a scalable solution for exploiting large amounts of unlabeled multimodal data. However, existing methods either...

Carlo Sgaravatti, Daniele Kopyshevskiy, Davide Secco et al. · 0 citations
Sep 2026

OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute

Feed-forward 3D reconstruction models have achieved impressive performance by scaling model and dataset size, but their cost excludes most research groups and precludes edge deployment. Additionally, generating 3D supervision without sensors still relies on slow, unreliable Structure-from-Motion, as the community lacks...

Brandon Leblanc, Charalambos Poullis · 0 citations
Book Open access Nov 2026

Beyond Sensing Fragility: Complementary-Pathway Distillation for Robust Depth Estimation

Vision foundation models have achieved remarkable success in monocular depth estimation, yet they tightly couple semantic and structural cues within a single RGB observation. This coupled representation makes them fragile to real-world imaging degradations such as motion blur and exposure failure, and inherently incomp...

Xiang Chen, Ya-Peng Meng, Yu-Guo Chen et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.