Beamforming Design for Cellular Coexistence With Incumbent Satellite Terminals in the Upper Mid-Band

Abstract

The upper mid-band, also known as frequency range 3 (FR3), spanning 7-24 GHz, has recently attracted significant interest for sixth-generation (6G) cellular systems due to its favorable balance between coverage and capacity. However, several portions of the FR3 spectrum are already occupied by incumbent satellite services, making spectrum coexistence between terrestrial cellular networks and satellite systems a key challenge. In particular, terrestrial base stations (BSs) must deliver high data rates to terrestrial user equipments (TUEs) while protecting satellite users, such as very small aperture terminals (VSATs), from harmful interference. In this paper, we investigate the coexistence between downlink terrestrial cellular networks and satellite users in FR3 through linear precoder optimization at a multi-sector BS. Specifically, we formulate and solve a terrestrial sum rate maximization problem subject to minimum rate requirements at the TUEs and maximum interference-to-noise ratio (INR) constraints at the VSATs across all sectors of a cell. To validate the method, we conduct a detailed multi-user site-specific ray-tracing (RT) simulation in a suburban environment using Sionna RT. The proposed framework is evaluated under different FR3 carrier frequencies, BS antenna array sizes, minimum rate requirements, INR thresholds, and VSAT densities. Simulation results demonstrate that the proposed approach significantly outperforms the existing spatial nulling benchmark, achieving a better tradeoff between terrestrial sum rate, TUE fairness, and incumbent VSAT protection. Furthermore, the results reveal that larger BS antenna arrays substantially improve coexistence feasibility and performance, whereas dense VSAT deployments can fundamentally limit simultaneous spectrum sharing. These findings provide practical design guidelines for enabling efficient downlink spectrum coexistence between future terrestrial cellular networks and incumbent satellite terminals operating in the FR3.