Piyush Kaushik Bhattacharyya

AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating

This paper proposes AutoNorm-S (Stabilized), a training strategy that mitigates optimization instability through a gate-freezing schedule, and suggests that decoupling normalization selection from optimization noise provides a practical and principled approach for adaptive normalization in Transformer architectures.