Routing Is Not Enough: Diagnosing Intra-Adapter Subspace Contention in MoE+LoRA Fine-Tuning
SpawnLoRA is evaluated on Phi-tiny-MoE-instruct and OLMoE-1B-7B across multiple mixture settings and finds that it effectively reduces negative transfer compared with standard and rank-adaptive LoRA, demonstrating that structural separation inside experts provides benefits beyond routing or rank expansion alone.
Mehreen Hossain Chowdhury, Nowshin Mahjabin, Ahmed Shafin Ruhan et al.
· 0 citations