AI-Driven Federated Learning Optimization for Privacy-Preserving Big Data Environments

Abstract

The fast growth of big data, along with the growing stricter privacy laws, poses tremendous challenges to the traditional centralized implementation of AI models. Federated Learning (FL) offers a decentralized solution; however, it is undermined by a number of performance limitations such as large communicational overhead, statistical heterogeneity due to nonindependent and identically distributed data, and small computational capability of edge clients. This paper presents an optimized AI-based FL architecture, which consolidates a client selection algorithm (that is based on reinforcement-learning) and a dynamic and fairness-conscious aggregation protocol, as well as a new query-fragment caching solution. The combined model is strategically focusing on high utility players, reducing skew in data during model aggregation, and lessening on-device unnecessary processing. The empirical analysis of CIFAR-10 and FEMNIST shows that the presented framework reduces the communication rounds by 38% and the model accuracy is increased by 5.7% when compared with Fed Avg and Fed Prox, at the same time, data privacy is also rigorously maintained.