An Operation-Aware Multimodal Perception Framework for Tugboat Assistance via Ego-Motion-Compensated AIS–Vision Fusion

Abstract

With the development of intelligent ports and autonomous maritime systems, reliable perception in complex tug-boat operation environments has become increasingly important for safe and efficient maritime operations. However, tugboat scenarios involve dense vessel interactions, frequent maneuvering behaviors, and dynamic environmental conditions, making single-source perception insufficient. Vision-based methods are affected by target occlusion, illumination variation, and small-scale vessels, while AIS-based methods suffer from low sampling frequency, packet loss, and asynchronous observations. To address these challenges, this paper proposes a multimodal maritime perception framework for tugboat operations by integrating AIS information and vision-based vessel trajectories. The framework first establishes a unified representation of heterogeneous observations through trajectory extraction, coordinate transformation, and cross-modal association. Ego-motion compensation is applied to reduce projection errors caused by tugboat translation and rotation. A physics-constrained AIS trajectory reconstruction method is then developed to recover continuous vessel motion from incomplete AIS measurements using vessel motion constraints and cubic spline interpolation. Furthermore, a two-stage AIS-vision trajectory matching strategy combines persistent association, spatio-temporal optimization, and geometric constraint matching to improve the accuracy and stability of heterogeneous trajectory fusion. Experiments on real-world tugboat operation scenarios demonstrate that the proposed framework improves the continuity and reliability of multi-source maritime perception under asynchronous observations and partial occlusion conditions, providing effective perception support for intelligent tugboat operation monitoring.