Hidden Gauge Controls Feature Specialization in ReLU Networks
This work shows that changing the scale of initial weights can determine which neurons learn a feature without altering any neuron's initial contribution, and reveals how initialization can control the organization of a learned representation without changing what the network initially represents.
Tong-Xi Wang
· 1 citation