Slow-Fast Multi-Teacher On-Policy Distillation (SF-MOPD), which couples a fast model, the current student updated directly by each teacher, with a slow model, an exponential moving average of the student, which effectively mitigates capability interference, enhances specialized multimodal capabilities, and reduces the...
Xiao-Fei Yin, Tong Chu, Ji-Yuan Fu et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.