Integrating over-the-air computations into the model aggregation process of federated learning (FL) offers a promising solution to mitigate the communication bottleneck in FL model training. In this approach, all the clients modulate their intermediate parameters, such as gradients, onto the same set of orthogonal waveforms and transmit the resulting signals to the edge server simultaneously. Capitalizing on the superposition property of the radio channel, the server can extract an automatically aggregated global gradient from the received radio signal. However, the limited number of orthogonal waveforms imposes a constraint on the dimensionality of transmittable updates, hindering the adoption of more advanced, but high-dimensional models. In light of this challenge, we propose OFLight, a lightweight, yet effective, gradient compression algorithm tailored for OTA-FL systems. Specifically, in each communication round, the edge server constructs a low-rank projection matrix based on the received gradient matrix from the previous round (initialized with an independent and identically distributed standard normal matrix in the first round) and broadcasts it, along with the global model, to all clients in the system. Based on this matrix, every client projects its locally updated gradient matrix into a low-dimensional subspace through a linear operation. The clients upload only their compressed gradients via OTA computations, and the edge server can perform a linear decompression on the received signal, retrieving the original gradient dimension. Moreover, an error feedback mechanism is incorporated to compensate for the approximation error under aggressive compression. We derive analytical expressions for the convergence rate of both convex and non-convex loss functions, quantitatively demonstrating the effect of OFLight on the OTA-FL training efficiency. We also conduct extensive experiments to corroborate the efficacy of the proposed method.
Jiaqi Zhu, H. Yang, Nikolaos Pappas et al.· IEEE Transactions on Wireles...· 0 citations
Federated learning (FL) applications normally employ large deep learning (DL) models, resulting in excessive communication overhead in the deployment of FL over resource-constraint mobile edge networks. To achieve better scalability for DL-based FL, we capitalize on both the asymmetric nature of mobile networks and the distinct effects of partial transmissions on FL training for the global and local models. We propose Fed-DynAmal, an FL framework that decreases the number of parameters transmitted in the uplink (clients-to-server) while concurrently achieving better model performance. The underlying idea is that each selected client sends a partial DL model to the server by omitting several sub-blocks from the trained local model. Crucially, we drop the assumption that transmitted local models can still be used for inference, thereby allowing for greater model variability. At the server, we introduce amalgamation, a process to merge different partial local models into an inference-viable full model. Essentially, amalgamation is a bridge for performing aggregation at the sub-block level. Interestingly, as the key takeaway, communication efficiency versus model performance is not necessarily a trade-off in FL: Our extensive experiments show that Fed-DynAmal can effectively improve communication efficiency while still concurrently achieving higher accuracy and enhanced robustness.
Zihan Chen, H. Yang, Tony Q. S. Quek et al.· IEEE Transactions on Cogniti...· 0 citations