A Novel Weight Initialization Scheme for Randomized Leaky Rectified Linear Units and a Comparative Analysis of Rectified Activation Functions for Image Classification

Abstract

The choice of activation functions and weight initialization strategies plays a critical role in the trainability and performance of deep neural networks. In deep architectures, improper weight initialization can lead to vanishing or exploding signals and gradients, resulting in unstable training behavior. This study proposes a novel weight initialization scheme specifically designed for the randomized leaky rectified linear unit (RReLU) activation function, with the objective of preserving signal statistics during both forward and backward propagation. The proposed formulation accounts for the stochastic distribution of the slope in the negative input region of RReLU when deriving variance-preserving initialization conditions for both forward and backward propagation. The method is theoretically derived and experimentally evaluated on feedforward neural networks and convolutional neural networks (CNNs). Comparative evaluations against the widely used Xavier and He initialization schemes demonstrate that the proposed approach provides more stable optimization and facilitates successful convergence, particularly as network depth increases. In addition, rectified activation functions—including the rectified linear unit (ReLU), leaky rectified linear unit (LReLU), parametric rectified linear unit (PReLU), and RReLU—are systematically compared using their corresponding initialization strategies on the CIFAR-10, CIFAR-100, and ImageNet datasets. The experimental results show that ReLU remains the most computationally efficient activation function, whereas PReLU achieves the highest classification accuracy on the large-scale ImageNet dataset. On the relatively smaller CIFAR-10 and CIFAR-100 datasets, LReLU and RReLU exhibit competitive generalization performance. These findings highlight the importance of activation-specific initialization strategies for enabling fair comparisons and improving the performance of deep learning models.