Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models
EoupCT effectively preserves both task-specific proficiency and inherent general-purpose knowledge, successfully mitigating the catastrophic forgetting, and forms a multi-objective optimization problem and introduces a first-order efficient Pareto optimizer.
Bing Wang, C. Li, Xin-Qiang Cai et al.
· 1 citation