Isotropic Activation Functions Enable Deindividuated Neurons and Adaptive Topologies

Abstract

Varying the number of neurons while preserving function is difficult because neural networks' dense connectivity creates strong computational interdependence between units. Elementwise activation functions, particularly their limited equivariance properties, are identified as a central hurdle to achieving this. We propose activation functions designed around chosen equivariances, enabling dense neural networks to adapt via singular value decomposition of their affine maps. This diagonalises layers into one-to-one, ordered connections, making it easier to assess the impact of individual connections on function. Low-impact neurons can be removed (neurodegeneration), and a thresholded buffer of largely inactive `scaffold' neurons is maintained (neurogenesis). These symmetry-led diagonalisation and structural changes prioritise function-invariance: neurogenesis exactly preserves function, neurodegeneration is asymptotically function-preserving with the pruned singular-value threshold. Moreover, this construction enables asymptotic 50\% parameter sparsification of dense networks while preserving function analytically. Thus, the approach enables real-time restructuring of the architecture in response to task demands, task appending, removal or changes. Conceptually, this motivates broader applications of symmetry prescriptions at the small scale, even when no data-driven model-scale symmetry adherence is required. Following this, we build activations around symmetry, not from classical assumptions about individual neurons. Unlike typical elementwise forms, these alternatives feature explicit basis independence and do not individuate neurons. We leverage this to decompose and reinterpret neurons across sequential layers, simplifying structure and providing a natural route to adaptive topologies. Additional developments include a new tunable parameter, the `intrinsic length'.