This paper proposes AutoNorm-S (Stabilized), a training strategy that mitigates optimization instability through a gate-freezing schedule, and suggests that decoupling normalization selection from optimization noise provides a practical and principled approach for adaptive normalization in Transformer architectures.
Abstract
Normalization is a critical component for stabilizing Transformer training, yet the choice between static strategies such as Layer Normalization (LN) and adaptive alternatives remains largely task-dependent. In this paper, we investigate a key optimization challenge in differentiable normalization gating. Our experiments show that, on relatively stationary vision tasks, the high gradient variance introduced by Gumbel-Softmax gating can hinder convergence of the routing mechanism, causing learned gates to underperform simple random selection. In contrast, on non-stationary language modeling and classification tasks, sustained gating diversity enables the model to learn more effective layer-wise normalization policies. Motivated by these observations, we propose AutoNorm-S (Stabilized), a training strategy that mitigates optimization instability through a gate-freezing schedule. AutoNorm-S achieves competitive or improved performance across multiple benchmarks, outperforming adaptive normalization baselines on NLP datasets, including PTB and SST-2, while remaining competitive on standard vision benchmarks. These results suggest that decoupling normalization selection from optimization noise provides a practical and principled approach for adaptive normalization in Transformer architectures.
Deep neural networks often contain substantial parameter redundancy, resulting in unnecessary computational cost and energy consumption. This work presents a dynamically adaptive gating mechanism for learning layer-wise sparsity through differentiable masking. In the proposed framework, each network parameter is associated with a self-learning gate that controls its contribution during training. The gating function follows a progressive soft-to-hard transition in which the slope is gradually annealed, enabling the model to move smoothly from continuous parameter weighting to near-binary pruning decisions while simultaneously learning an adaptive threshold parameter. The framework is evaluated across multilayer perceptrons (MLPs), deep neural networks (DNNs), Tabular Transformer models, and benchmarked against $\mathrm{L}_{0}$ regularization and Variational Dropout on convolutional neural networks. Experimental results demonstrate substantial model compression without degrading predictive performance, and in some cases improving it. On MNIST, the gated MLP retains only 14% of weights (86% pruning) while maintaining 98% accuracy. The gated DNN and Tab Transformer similarly outperform their dense counterparts while retaining only 46% and 46.22% of parameters, respectively. Benchmarking further shows that the proposed adaptive gating achieves competitive or superior accuracy-sparsity trade-offs compared with $\mathbf{L}_{0}$ regularization and Variational Dropout. These results demonstrate that the proposed dynamically adaptive gating framework provides an efficient and interpretable pathway for sparsity learning, enabling high-performing lightweight neural network deployment.
Raunak Dev, Mydhily Sankar, Devaprabha Biju S et al.· 2026 International Conferenc...· 0 citations
Deep learning models lack intrinsic interpretability and require manual tuning of information-theoretic hyperparameters. This paper presents HIFN-Transformer (HIFN-T), a framework extending the Variational Information Bottleneck through four jointly learnable per-layer parameters: information retention, entropy budget, magnitude scaling, and global information gates. The retention and budget parameters are co-adapted with the network weights via a stop-gradient decoupling that stabilizes training and substantially reduces the manual hyperparameter burden by eliminating per-layer $\beta $ grid search; the global gates learn per-neuron importance weights, and a Transformer encoder captures complex feature interactions via multi-head self-attention. We prove HIFN generalizes standard VIB as a special case and characterize the role of the entropy budget as an early-training stabilizer that prevents variance explosion before convergence. We evaluate HIFN-T on five benchmarks spanning network intrusion detection (CICIDS-2017, NSL-KDD, GothamDataset2025), image classification (MNIST), and speech emotion recognition (EMO-DB), comparing against XGBoost, TabNet, FT-Transformer, and four additional baselines. HIFN-T delivers consistent improvements across all five. On the official NSL-KDD KDDTrain $+ \,\, \rightarrow $ KDDTest+ benchmark, HIFN-T with a 4-method consensus feature-selection front-end achieves 80.22% accuracy and 0.6898 macro F1, lying within the established 78–87% band of published deep models on this cross-distribution split. On the large-scale GothamDataset2025 IoT benchmark, we evaluate HIFN-T on the full 35,126,611-row release under natural class prevalence as a scalability demonstration on this recently published dataset. MNIST is included as a domain-generalization sanity check rather than a competitive benchmark. Gate-derived feature importance correlates strongly with SHAP analysis across all five datasets, validating intrinsic interpretability. We report all results as mean ± standard deviation over ten independent runs with Friedman and Wilcoxon signed-rank statistical significance tests.
Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on. We introduce AOS-R (Adaptive Optimizer Switching, Rule-Based), a lightweight controller that monitors six online gradient-space signals -- gradient noise scale (GNS), Hutchinson curvature trace, loss stagnation, update stability ratio, gradient stability index (GSI), and loss improvement ratio (LIR) -- and switches among AdamW, SGD-M, and Lion as the optimization landscape evolves. State-preserving momentum transfer and a 400-step learning-rate bridge prevent accuracy degradation at every transition point. On CIFAR-100/WRN-28x10, AOS-R reaches 78% top-1 in 81 epochs -- 26% fewer than AdamW (109), 43% fewer than SGD-M (143), and 16% fewer than Lion (96). Across eight model-dataset benchmarks, AOS-R achieves best accuracy on 6 of 8 combinations with a mean +0.4 pp gain and 0.80x convergence speedup over AdamW under a single shared hyperparameter configuration.
A. K. Pandey, Umang Chaturvedi, Aatish Rana et al.· 0 citations
Activation functions are considered an essential primitive for neural nonlinearity, i.e., they enable neural networks to serve as universal approximators. In this paper, we show that this nonlinearity can also be achieved by input-conditioned threshold gating through branches as a universal primitive. We demonstrate that standard activations -- whether piecewise-linear (ReLU, PReLU, Hardtanh) or smooth (SiLU, Sigmoid, Tanh, GELU) -- are in fact instances of a single Threshold Gating (TG) primitive. For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form. We then validate these equivalences by converting pretrained networks across CNNs, transformer-based models, and recurrent architectures, preserving model performance without requiring retraining. Threshold Gating also enables training from scratch that goes beyond replacing existing activations, enabling gains in model compression, performance, and shorter training. We also propose a'Minimal Branch Theorem'which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks. In terms of hardware implementation, TG maps to a unified implementation in the case of analog in-memory systems, addressing the bottleneck of analog-to-digital and digital-to-analog converters (ADC/DAC) that is known to significantly impact power consumption and on-chip area.
Muhammad Sabih, Frank Hannig, Jürgen Teich· 0 citations
Deep ensembles provide the most reliable uncertainty estimates in deep learning, but their cost grows linearly with the number of members. Implicit ensembles lower this cost by sharing a single backbone across members. Member diversity is a primary determinant of ensemble quality, yet no implicit ensemble can shape it during training; existing methods fix it at initialisation or build it into the architecture. We introduce $\sigma$N-Ens, a normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture and modulates the shared backbone through sigmoid-bounded scalers. We also introduce a softmax-temperature regulariser, which shapes the equilibrium level of sharing between members and traces the accuracy-calibration frontier. Because only normalisation layers are replicated, the mechanism can wrap convolutional and transformer backbones alike, also allowing pretrained models to be adapted through a short fine-tune. We frame the epistemic uncertainty such an ensemble expresses as modulation uncertainty, and explain why its calibration holds under input corruption, and why its out-of-distribution detection is weaker. Our method is evaluated across ResNets and transformers on CIFAR-10/100, ImageNet and SST-2. $\sigma$N-Ens matches or outperforms deep ensembles at a fraction of their parameter cost, scales with ensemble size where partitioning methods collapse, and maintains calibration under distribution shift.
Parameter-Efficient Fine-Tuning (PEFT) has become the de facto standard for adapting Vision Transformers (ViTs) to downstream tasks. While parameter count has been the dominant efficiency metric in PEFT, it does not imply \textit{compute efficiency}: parameter-sparse methods can still incur full-model training cost per step, and typically need long schedules to reach peak accuracy. We introduce Circuit Fine-Tuning (CFT), a compute-efficient framework that uses circuit discovery---conventionally used to explain trained models---to select modules for fine-tuning before training. Whereas attribution is conventionally formulated against a trained task head, we formulate it against a near-zero-initialized probe head, which isolates the response of the backbone to the target distribution rather than the preferences of a particular classifier. CFT then fine-tunes only the recovered subgraph. CFT needs no learning-rate warmup and reaches peak accuracy in ${\sim}20$ epochs on average---versus $44$--$96$ for strong PEFT baselines---yielding $2.3$--$6.6\times$ fewer training FLOPs and up to $16\times$ less wall-clock time, while adding zero parameters and no inference operations. Experiments across a standard visual transfer benchmark (VTAB-1k), hierarchical backbones (Swin), domain-shifted medical imaging (CBIS-DDSM), and a vision-language model (Gemma-3 on CUB-200) demonstrate the effectiveness of CFT. Code is available at https://github.com/UriKialy/CFT