Aug 2026· International Conference on Machine Vision and Deep Learning· Vol 14326, pp. 1432630 - 1432630-6· 0 citations· 13 references
Engineering
TL;DR
PEFT-Assess is presented, a systematic framework for evaluating the adaptability of four mainstream PEFT methods—LoRA, QLoRA, Adapter, and Prefix-Tuning—across five evaluation dimensions: accuracy, F1-score, training speed, memory efficiency, and generalization capability.
Abstract
Parameter-efficient fine-tuning (PEFT) methods have become essential for adapting large language models (LLMs) to downstream tasks without updating all parameters. However, the optimal PEFT strategy varies significantly across model architectures, task types, and resource constraints, making method selection a non-trivial problem. This paper presents PEFT-Assess, a systematic framework for evaluating the adaptability of four mainstream PEFT methods—LoRA, QLoRA, Adapter, and Prefix-Tuning—across five evaluation dimensions: accuracy, F1-score, training speed, memory efficiency, and generalization capability. We benchmark these methods on LLaMA-7B and Qwen-7B across six NLP tasks from the GLUE and SuperGLUE benchmarks. Furthermore, we propose a gradient boosting-based strategy selector that predicts the optimal PEFT method given task characteristics and resource budgets. Experiments show that LoRA achieves the best overall adaptability score of 0.847, while our strategy selector attains 91.2% prediction accuracy for recommending the optimal PEFT method.
Large language models have demonstrated outstanding performance in the field of natural language processing, but pretrained models are difficult to adapt to specific tasks. Fine-tuning technology has become a key solution, but its application is limited by problems such as complex parameter selection, high training costs, and uncertain effect evaluation. This study aims to construct a complete framework for optimizing fine-tuning parameters of large language models, to achieve accurate prediction and scientific evaluation of fine-tuning effects, and to solve the problems of overfitting and high-performance adaptation under limited data. By constructing a multi-task mixed dataset, comparative experiments were designed using the controlled variable method. The influence of key hyperparameters such as learning rate and batch size was systematically analyzed, the performance of classic and modern optimization algorithms was compared, and a prediction mechanism was established by combining regression and classification models. A comprehensive evaluation system based on "performance-efficiency-resource" was constructed. The results show that a learning rate of 1E-3 and a batch size of 64 are the optimal configurations. Modern optimization algorithms converge faster, and multinomial regression and neural networks perform best in effect prediction. The performance of small-scale models after fine-tuning can surpass that of un-fine-tuned large models, significantly reducing computational costs. This study provides a scientific basis for optimizing fine-tuning parameters, improves resource utilization efficiency, and promotes the efficient implementation of large language models in vertical fields.
Yi-He Ding· International Conference on...· 0 citations
Performance consistently improves from conventional single-policy PEFT to multi-policy PEFT, with the proposed automatic multi-policy framework achieving the best performance under the same trainable capacity, suggesting that optimization-path organization is more effective than simply increasing adapter capacity for heterogeneous parameter-efficient fine-tuning.
Jia-Jia Tang, Si-Zhe Yuen, Francisco Gomez Medina et al.· arXiv.org· 0 citations
This work proposes eXtreme Gradient Boosting LoRA (XGBLoRA), a novel framework grounded in gradient boosting theory that provides theoretical analysis establishing convergence guarantees and expressiveness bounds, which formally justify why weaker (lower-rank) adapters, when properly combined, can match or exceed the performance of stronger ones.
Yifei Zhang, Hao Zhu, Haoran Shi et al.· Proceedings of the 32nd ACM...· 0 citations
Parameter-efficient fine-tuning (PEFT) methods adapt foundation models to specific domains by selectively updating only a small subset of critical parameters, significantly reducing computational costs. Among these methods, Sparse Low-Rank Adaptation (SoRA) has emerged as an effective approach for fine-tuning large language models. This paper investigates SoRA algorithm's performance in large-scale models, particularly analyzing learning dynamics and scaling effects in wide neural networks. We find that employing a uniform learning rate across all parameter matrices limits learning efficiency. To address this, we derive an improved learning rate configuration and propose SoRA+, an enhanced variant that introduces a hyperparameter to adjust learning rates proportionally. This modification aligns with theoretical optima and mitigates the suboptimality of the original SoRA method. Experiments show that SoRA+ improves fine-tuning performance by up to 2% while accelerating convergence.
The current robustness fine-tuning methods improve over standard fine-tuning and in-context learning, but the best-to-worst prompt gap remains as high as 40-57% of performance, and recent robustness-enhancing methods often fail to outperform the simplest data construction strategy: training on one template per batch.
This work proposes StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility, and experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions.
Jing-Han Tan, Yuanzhe Wang, Lu Chen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
