When Gradient Boosting Meets Adapter: Exploring Weak Learners for Parameter-Efficient Fine-tuning of LLMs

Abstract

Fine-tuning Large Language Models (LLMs) has become a crucial technique for adapting pre-trained models to downstream tasks. However, the enormous size of LLMs poses significant challenges in terms of computational complexity and resource requirements. Low-Rank Adaptation (LoRA) has emerged as a promising solution, yet a gap remains between the practical performance of low-rank adaptations and their theoretical optimum. While recent works have explored iteratively merging LoRA adapters, they lack a principled theoretical framework to guide adapter design. In this work, we propose eXtreme Gradient Boosting LoRA (XGBLoRA), a novel framework grounded in gradient boosting theory. Our key insight is that the adapter must adhere to the weak learner principle-each individual adapter should have limited expressiveness---to ensure that the iterative ensemble can effectively raise the model's performance ceiling without overfitting. We provide theoretical analysis establishing convergence guarantees and expressiveness bounds, which formally justify why weaker (lower-rank) adapters, when properly combined, can match or exceed the performance of stronger (higher-rank) ones. Extensive experiments on natural language processing tasks demonstrate that XGBLoRA with rank-1 updates consistently outperforms standard LoRA with significantly fewer trainable parameters.