Efficient Posterior Approximation for Bayesian Neural Networks in Large-Scale Distributed Environments

Abstract

Bayesian neural networks (BNNs) introduce a principled framework for uncertainty estimation through learning probability distributions with respect to model parameters. Still, current deep networks are not only unfeasible to perform exact Bayesian inference, but traditional posterior approximation methods also incur significant computing, memory, and communication costs in large-scale distributed systems. This paper introduces an efficient distributed posterior approximation framework for scalable Bayesian neural network training. The framework combines structured variational inference, asynchronous distributed optimization, and communication-efficient gradient compression in order to lower the costs of posterior estimation across multiple computing nodes. Notably, in contrast to traditional mean-field approximations, their proposed method retains critical posterior relationships while ensuring computational efficiency. For asynchronous updates, the synchronization delays due to heterogeneous workers are minimized, while compressed parameter communication decreases network overhead during distributed training. The presented approach yields better predictive accuracy, uncertainty calibration, and scalability for the observed parameters for different worker configurations and data distributions when compared to existing Bayesian inference methods and is shown through experimental evaluation. By achieving an accuracy of 87.4% and reducing costly communication and training time, the framework also performs consistently under independent and identically distributed (IID) and non-IID conditions. These results show that the proposed framework could offer effective and scalable posterior approximation for Bayesian neural architectures used in large-scale distributed environments.