How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
The studied model realizes only part of the flexibility afforded by four-stream mHC: individual blocks rarely require all four streams, and late residual mixing provides little measured benefit.
Peng-Xiang Zhao, Xing Li, Xian-Zhi Yu et al.
· 1 citation