Rethinking Contextualization by Reinterpreting Attention Head Channels
Contextualization, the core operation of language modeling, transmits information across words to build sentence-specific word representations. Prior works mainly study contextualization, focusing on individual words and attention heads as a growing discrete dictionary, lacking a global view of their general behavior....
Hakaze Cho, Hao-Lin Yang, Zhun Sun et al.
· 0 citations