Benjamin Piwowarski

Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

Building on evidence that positional and semantic signals occupy nearly orthogonal subspaces in trained Transformers, an encoder Transformer is modified to process three explicitly disentangled streams: semantic, absolute positional (AP) and relative positional (RP), and confine the masked-language-modeling (MLM) objec...