From symmetry to sparsity: A dual framework of LiSHT and ReXTanh activation functions for deep learning
TL;DR
This paper analyzes the already established Linearly Scaled Hyperbolic Tangent function and proposes ReXTanh, a smooth-sparse hybrid function that provides C 1 continuity while remaining asymmetric, showing that keeping negative magnitudes can outmatch forced sparsity in harder recognition problems.
Abstract
. Activation functions are one of the fundamental concepts in the field of artificial neural networks used to optimize their performance and feature extraction. In this paper, we introduce a dual approach to those non-linear transformations, focusing on the balance between keeping information and enforcing sparsity. We start by analyzing the already established Linearly Scaled Hyperbolic Tangent (LiSHT) function. Our mathematical proofs demonstrate that it provides a smooth and differentiable approximation of the absolute value function. This makes LiSHT a differentiable L1-like regularizer that does not discard negative signals. Next, for models that need strict sparsity, we propose ReXTanh, a smooth-sparse hybrid function that provides C 1 continuity while remaining asymmetric. We tested both functions extensively on HPC clusters using mixed-precision distributed training, and the results highlight a clear split in use cases. While ReXTanh works best as a sparse operator for standard image datasets, yielding higher accuracy, LiSHT’s symmetric shape handles complex and noisy data much better, showing that keeping negative magnitudes can outmatch forced sparsity in harder recognition problems.