Multiple Modalities Image Matching With Large-Scale Pre-Training.
TL;DR
A large-scale pre-training framework that utilizes synthetic cross-modal training signals, incorporating diverse data from various sources, to teach models to recognize and match fundamental structures across images, which generalizes effectively across more than eight unseen cross-modality registration tasks.
Abstract
Image matching, which aims to identify corresponding pixel locations between images, is crucial in a wide range of scientific disciplines, aiding in image registration, fusion, and analysis. However, when dealing with images captured under different imaging modalities that result in significant appearance changes, the performance of learning-based image matching algorithms often deteriorates due to the scarcity of annotated cross-modal training data. This limitation hinders applications in various fields that rely on multiple image modalities to obtain complementary information. To address this challenge, we propose a large-scale pre-training framework that utilizes synthetic cross-modal training signals, incorporating diverse data from various sources, to teach models to recognize and match fundamental structures across images. This capability is transferable to real-world, unseen cross-modality image matching tasks. Our key finding is that the matching model trained with our framework generalizes effectively across more than eight unseen cross-modality registration tasks using the same set of network weights, substantially outperforming existing generalizable methods and achieving competitive or superior performance compared to specialized models on several tasks.