SIGMA: Statistical Memory Head for Robust Class-Incremental Vision–Language Learning with CLIP

Abstract

Class-incremental learning with vision–language models faces a fundamental tension: while CLIP provides strong zero-shot generalization, fine-tuning task-specific adapters induces representation drift and biases decision boundaries toward recent classes, leading to catastrophic forgetting. Freezing the CLIP backbone alleviates drift but does not resolve residual prediction bias, and existing adapter-based methods leave the stable raw CLIP feature space largely unexploited for logit recalibration. To address these limitations, we propose SIGMA (Statistical Inference Gaussian Memory Adaptation), a training-free inference module that accumulates per-class Gaussian statistics directly in the frozen CLIP feature space. SIGMA performs analytic discriminant classification to recalibrate predictions without using gradient updates, eliminating the overfitting risk inherent in adapter-based optimization. When integrated with lightweight adapters, SIGMA strikes an effective balance between short-term adaptation and long-term stability, addressing both representation drift and prediction bias within a unified inference-time framework. Experimental results on CIFAR-100, ImageNet-R, and CUB-200 demonstrate consistent improvements compared to state-of-the-art continual learning methods.