DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integration
This work meticulously design a dual-stage pretraining work flow with four dedicated submodels to capture representative dependency agreements at the chunk level, effectively transforming these agreements into embeddings to benefit the pretraining.
Martin Kuo, Jian-Yi Zhang, Dong-Ting Li et al.
· 2 citations