Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Weight Quantization.

Abstract

The proliferation of artificial neural networks (ANNs) has led to increased energy consumption, raising concerns about their sustainability. Spiking neural networks (SNNs) operate using sparse, binary spikes to communicate information between neurons and offer a potential solution due to their limited energy requirements. Another technique for reducing a neural network's footprint is quantization, which compresses weight representations to decrease memory usage and energy consumption. In this study, we present twin network augmentation (TNA), a novel training method aimed at improving the performance of SNNs on a range of benchmark data sets while also facilitating enhanced network compression through quantization of weights. TNA involves cotraining an SNN with a twin SNN with identical network architecture, optimizing both networks to minimize their cross-entropy losses and the mean squared error between their output logits. We demonstrate that TNA significantly enhances classification performance across various data sets and can be applied for reducing SNNs to ternary weight precision for inference. Our results show that TNA outperforms traditional knowledge distillation methods and achieves state-of-the-art performance for the evaluated network architecture on benchmark data sets, including CIFAR-10, CIFAR-100, and CIFAR-10-DVS. This letter underscores the effectiveness of TNA in bridging the performance gap between SNNs and ANNs and suggests further exploration into the application of TNA on different network architectures and data sets.