Optimizing knowledge distillation models for language models

Abstract

The problem of deep neural network compression is discussed using Convolutional Neural Networks (CNNs) as an example. The size of deep neural networks is an obstacle to their practical application under conditions of limited computing resources, energy, and inference latency requirements. One of the developing approaches to compressing deep neural network models is thinning-removing some of the parameters or structural elements of the neural network model. It is shown that thinning is a tradeoff between classification accuracy and computational efficiency. The problem of compressing CNN models was formulated by thinning parameters while maintaining classification quality at the level of the original model, reducing the number of parameters, computational complexity, and inference latency. Standard Top-1 and Top-5 classification accuracy metrics were used to evaluate classification quality. The degree of compression and inference latency were assessed using metrics, such as the number of model parameters, model size, computational complexity, and inference latency. To ensure a fair comparison of the proposed method with others, inference latency measurements were conducted under identical conditions with a fixed input data size. The concept of representation geometry is introduced to control the stability of the internal feature space. The change in the interclass similarity matrix, calculated from the class centroids in the feature space, is used as a metric for preserving the structure of the feature space. The main result of this work is the development of a new compression method for CNNs based on geometrically controlled thinning of the CNN model. In this method, thinning is performed greedily across CNN blocks, and the admissibility of each step is determined not only by the local importance of channels but also by a global constraint on changing the geometry of the representations. The results of the experiment demonstrate that the proposed method consistently maintains classification quality while reducing computational complexity and the number of parameters compared to the baseline method without thinning and the depth-based thinning method of the CNN model. The proposed method for geometrically controlled thinning of a CNN model can be applied to problems requiring real-time decision making as well as on mobile and embedded devices.