Han Yu

Fisher-Based Layer-Wise Adaptive Sparsification for Efficient Pruning of Large Language Models

When Gradient Boosting Meets Adapter: Exploring Weak Learners for Parameter-Efficient Fine-tuning of LLMs

Fine-tuning Large Language Models (LLMs) has become a crucial technique for adapting pre-trained models to downstream tasks. However, the enormous size of LLMs poses significant challenges in terms of computational complexity and resource requirements. Low-Rank Adaptation (LoRA) has emerged as a promising solution, yet a gap remains between the practical performance of low-rank adaptations and their theoretical optimum. While recent works have explored iteratively merging LoRA adapters, they lack a principled theoretical framework to guide adapter design. In this work, we propose eXtreme Gradient Boosting LoRA (XGBLoRA), a novel framework grounded in gradient boosting theory. Our key insight is that the adapter must adhere to the weak learner principle-each individual adapter should have limited expressiveness---to ensure that the iterative ensemble can effectively raise the model's performance ceiling without overfitting. We provide theoretical analysis establishing convergence guarantees and expressiveness bounds, which formally justify why weaker (lower-rank) adapters, when properly combined, can match or exceed the performance of stronger (higher-rank) ones. Extensive experiments on natural language processing tasks demonstrate that XGBLoRA with rank-1 updates consistently outperforms standard LoRA with significantly fewer trainable parameters.

Federated Nonlinear Causal Discovery via Divide-and-Conquer Learning

Federated causal discovery aims to learn causal structures from distributed data without sharing raw samples. Existing federated nonlinear methods adopt a monolithic global strategy that optimizes the entire graph simultaneously, suffering from catastrophic error propagation: a single misidentified edge cascades through the global structure, severely degrading accuracy under heterogeneous and limited local data. We propose DC-FNCD (Divide-and-Conquer based Federated Nonlinear Causal Discovery), which decomposes the global problem into independent per-variable neighborhood learning tasks. The core mechanism is a characteristic function-based conditional independence test whose empirical statistics admit exact linear decomposition across clients, enabling lossless federated aggregation without raw data exchange. Local neighborhoods are merged via conflict-aware skeleton construction and oriented through federated additive noise model testing. Extensive experiments demonstrate that DC-FNCD significantly outperforms state-of-the-art federated baselines. The source code is available at https://github.com/Xianjie-Guo/DC-FNCD.