What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use
What should data teach a language model at a particular point in training? A circuit view reveals three distinct bottlenecks: forming a computation, making its required content available, and selecting among available routes. A shared diagnosis-to-data principle connects them: localize the missing operation, preserve i...
Yi-Xiao Chen, Ke Cheng, Jiang-Tao Guan et al.
· 0 citations