Muon Can Outperform Dedicated Continual Learning Methods
This work trains a plain incremental LoRA (IncLoRA) with Muon, which orthogonalizes each update, and compares it against O-LoRA and ELLA over five seeds and three task orders on the Standard CL Benchmark and three seeds on TRACE.
Sebastian George Sincari, Bogdan Alexandru Gheorghe, Antonio Barbalau Faculty of Mathematics et al.
· 0 citations