Improving performance in SNN-based deep reinforcement learning via transition-aware embeddings

Abstract

Deep Reinforcement Learning (DRL), which integrates reinforcement learning with deep neural networks (DNNs), has been extensively researched across diverse domains. Robotics, in particular, has seen significant advancements, as DRL enables agents to extract meaningful features from high-dimensional observations and make precise decisions within complex environments. Furthermore, in real-world control tasks, achieving high performance within a limited number of interactions is required, particularly in complex and unknown environments. In addition, since real-world trials are constrained by cost and safety considerations, designing DRL algorithms that enable high sample efficiency and stable decision-making is of critical importance. Spiking neural networks (SNNs), inspired by biological neural systems, have recently attracted increasing attention due to the binary and event-driven nature of their information processing. Motivated by these properties, there has been growing interest in integrating SNNs into DRL. Nevertheless, SNN-based DRL faces challenges in learning performance. To address this issue, this study introduces embeddings learned from environmental dynamics into SNN-based DRL. Experimental evaluations conducted on four continuous control tasks from OpenAI Gym demonstrate that the proposed method outperforms conventional SNN-based DRL methods, both in terms of maximum average rewards and accelerated reward acquisition during the early stages of learning. These results establish that leveraging transition-aware embeddings effectively enhances both performance and sample efficiency in SNN-based DRL.