Feature Norm Normalization: A Plug-and-Play Module for Boosting Model Accuracy in Machine Unlearning

Abstract

Driven by stringent privacy regulations and data deletion requirements, machine unlearning has emerged as a critical field focused on selectively removing the influence of specific data from the pre-trained model. In this paper, we focus on achieving the unlearning objective while maintaining better model accuracy. Firstly, we identify and quantitatively characterize a previously overlooked cause of model accuracy drop during the unlearning process: feature norm shift. Then, to address this, we propose a simple yet efficient plug-and-play module, namely Feature Norm Normalization (FNN). Notably, our FNN can be seamlessly integrated into existing unlearning schemes to explicitly constrain the feature norm shift, and thus stabilize model accuracy. Extensive experiments also show that FNN can effectively help existing unlearning schemes achieve higher model accuracy. For instance, on CIFAR10 with ResNet18, using Salun with FNN for randomly unlearning 64 data achieves 50.89% higher accuracy than standard Salun.