Fast and Scalable Reliable-Transport for Cross-Datacenter Lossy RDMA Networks

Abstract

Extending lossy RDMA to cross-datacenter networks is increasingly important for distributed cloud services. However, existing lossy-RDMA reliability mechanisms are designed for intra-datacenter environments and suffer from three fundamental limitations when operating over long-haul links: (1) high feedback delay, (2) excessive end-host state overhead, and (3) lack of precise inflight packet estimation required by window-based congestion control. To systematically address these limitations, we propose FSRT, a novel reliable-transport mechanism for cross-datacenter lossy RDMA networks. FSRT introduces three key innovations. First, a dual-loop feedback mechanism leverages source-side Datacenter Interconnect (DCI) switches to generate near-source loss notifications, significantly reducing feedback latency. Second, a counting-based packet tracking mechanism replaces per-packet bitmaps with message-level lightweight counters, reducing NIC state overhead by nearly two orders of magnitude. Third, an inflight packet estimation mechanism enables accurate inflight packet counting, providing effective support for window-based congestion control. Simulation results show that FSRT reduces the average FCT for intra-datacenter and inter-datacenter traffic by up to 64% and 58%, respectively.