Dynamic structured pruning and merging for real-time acceleration of reinforcement learning.
TL;DR
By periodically merging parallel network instances, this method effectively counteracts the instability triggered by aggressive structural changes, enabling the use of structured pruning directly compatible with general-purpose hardware.
Abstract
Deep reinforcement learning (DRL) remains constrained by high computational costs, hindering its practical deployment. While neural network pruning offers a solution for model compression, most existing DRL approaches rely on unstructured pruning. This results in irregular sparse matrices that lack compatibility with standard hardware, offering no tangible real-time acceleration. Furthermore, pruning in DRL is notoriously challenging due to inherent training instability, which often leads to catastrophic performance degradation. To overcome these limitations, we propose a novel framework that integrates dynamic structured pruning with model merging. By periodically merging parallel network instances, our method effectively counteracts the instability triggered by aggressive structural changes, enabling the use of structured pruning directly compatible with general-purpose hardware. Experimental results on continuous control tasks demonstrate that our approach reduces cumulative training FLOPs by up to 72% while maintaining performance competitive with dense baselines across the majority of environments. Additionally, inference profiling confirms an average latency reduction of 16.2%, highlighting the framework's potential for accelerating both the training and deployment of DRL agents. Our code is available at https://github.com/hail-mary/neuron-pruning.