Few-Shot Class-Incremental Learning With Low-Rank Prototype Adaptation
Abstract
Few-Shot Class-Incremental Learning (FSCIL) aims to incrementally expand recognition capability with limited new samples while preserving previously acquired knowledge. Existing methods typically fine-tune the backbone or directly update class prototypes through global adjustments to accommodate new knowledge. These strategies tend to disturb the well-structured representation space formed by the base classes and degrade the essential knowledge encoded in the original prototypes. To address this issue, we freeze the backbone during incremental sessions and introduce a novel Low-Rank Prototype Adaptation (LRPA) framework for efficient semantic drift compensation in the prototype space. The main insight is that semantic drift can be explicitly modeled as a low-dimensional residual over the initial mean prototypes of novel classes, which captures the systematic bias induced by limited samples and enables effective adaptation without rewriting the original prototypes. LRPA consists of a Structural Alignment Projector (SAP) to enhance the generalization of the backbone and a Low-Rank Continual Residual Adapter (LRCRA) to incrementally adapt the model to new tasks. SAP leverages a pair of low-rank matrices in conjunction with the prototype classifier to explicitly mimic the subsequent model architecture, encouraging backbone alignment with the LoRA paradigm and emphasizing principal components for more discriminative prototypes. LRCRA involves a pair of low-rank matrices that independently retain task-specific information, serving as a compatible prototype for compensating semantic drift. Only a small subset of parameters requires updating, enabling LRCRA to efficiently adapt to novel classes without significantly altering the original prototype structure. Extensive experiments validate the superior performance of LRPA.