Multi-Task Dense Prediction with Mixture of Unit-Rank Experts

Abstract

Multi-task dense prediction has recently achieved significant performance gains but remains challenging in learning fine-grained and discriminative task-specific representations. In this paper, we propose a novel decoder-focused Mixture of Unit-Rank Experts (MURE) for multi-task dense prediction. Inspired by the subspace mixing property of LoRA, we introduce unit-rank experts with a LoRA-style structure into a mixture-of-experts (MoE) framework to facilitate adaptive fine-grained modulation of feature representations. Building upon this design, we develop spatial and channel modulation units in the decoder, which progressively refine task-specific features by capturing structural patterns and semantic relationships, respectively. Furthermore, to mitigate the interference between cross-task generic knowledge and task-specific information in multi-task learning, we employ a hybrid MoE design. Specifically, MURE incorporates both task-shared and task-private expert groups within each modulation unit to enable effective simultaneous modeling of inter-task general- and intra-task specialized representations, thereby promoting more discriminative task-specific features. Extensive experiments on the PASCAL-Context and NYUD-v2 benchmarks demonstrate that our MURE achieves superior overall performance compared to previous state-of-the-art methods.