A Heterogeneous Multiagent Reinforcement Learning Approach for Robust Uplink Beamforming in Maritime Satellite Communications

Abstract

Maritime satellite communications (SATCOMs) are expected to support high-capacity ship-to-satellite uplinks for remote maritime services beyond terrestrial coverage, with low-Earth-orbit (LEO) satellites providing wide-area connectivity. However, robust uplink beamforming in LEO maritime SATCOMs is challenging because dynamic ship–satellite geometry, wave-induced attitude motion, imperfect channel state information, and multiship interference make transmit power, ship-side transmit beamforming, and satellite-side receive combining tightly coupled. Accordingly, we formulate a long-term spectral efficiency (SE) maximization problem under transmit-power and quality-of-service constraints. An attitude-aware uplink channel model is developed by incorporating roll, pitch, and yaw motions into the effective angle-of-departure/angle-of-arrival evolution. Based on this model, the problem is cast as a heterogeneous decentralized partially observable Markov decision process. We then propose a robust heterogeneous cooperative QMIX (RHC-QMIX) framework under centralized training and decentralized execution, where type-specific recurrent local Q-networks, history-refined angular features, and centralized monotonic value mixing coordinate ship and satellite agents. Extensive simulations demonstrate that in the load-controlled scalability evaluation, RHC-QMIX achieves an average network SE of 25.32 bps/Hz, improves over alternating optimization by up to 51.00% as the satellite load increases, and outperforms heterogeneous cooperative QMIX by 16.38% on average under network-size scaling; it also maintains more stable SE under severe sea-state-induced ship motion.