Dense Image Matching Method Based on Transformer and Multi-Scale Feature Fusion
TL;DR
A dense matching network based on a Transformer and multi-scale feature fusion, called Task-aware Multi-Scale Matching Network (TMSMNet) is proposed, which outperforms mainstream methods such as RAFT-Stereo on the D1-all metric of KITTI- 2015 and demonstrates good generalization and robustness.
Abstract
Dense image matching is crucial in applications such as 3D reconstruction, autonomous driving, and remote sensing mapping; however, weak textures, occlusions, and large-disparity scenes remain challenging. To address these issues, this paper proposes a dense matching network based on a Transformer and multi-scale feature fusion, called Task-aware Multi-Scale Matching Network (TMSMNet). First, Swin Transformer is used to model global context in feature maps, enhancing the feature discriminability in weak texture regions. Then, a multi-scale cost volume is constructed, and adaptive fusion is achieved through deformable convolution to accommodate disparity variations of different ranges. Finally, an attention- guided iterative optimization module is introduced to improve the matching accuracy in occluded regions. Experimental results on the Scene Flow, KITTI-2015, and Middlebury datasets show that TMSMNet outperforms mainstream methods such as RAFT-Stereo on the D1-all metric of KITTI- 2015 and demonstrates good generalization and robustness. Ablation studies also confirm the effectiveness of each module. In summary, the method in this paper provides a feasible approach for dense matching. Future work will explore model lightweighting to support real-time applications and attempt to combine generative models to handle completely textureless regions, further enhancing its performance in complex scenes.