Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models
Typhoon, a masking strategy that uses the gradient of the MLM loss with respect to one-hot token inputs to estimate online how much each token type contributes to the objective, is introduced and evaluated against random masking and whole-word masking.
Bruce Changlong Xu, Muhammed Shahir Abdurrahman, Hashem Elezabi
· 0 citations