Deep Reinforcement Learning-Based Energy Management of Battery-Supercapacitor Hybrid Storage Systems in Renewable Microgrids

Abstract

The increasing integration of renewable energy sources into microgrids has intensified the need for intelligent energy management strategies capable of addressing the intermittency of solar and wind generation while ensuring reliable and cost-effective operation. Although Rule-Based Control (RBC) methods are straightforward to implement, their limited adaptability often leads to suboptimal utilisation of Hybrid Energy Storage Systems (HESS). This study develops and evaluates a Deep Reinforcement Learning (DRL)-based energy management system employing a Deep Q-Network (DQN) to coordinate battery–supercapacitor operation within a renewable microgrid. A Gymnasium-compatible simulation environment was constructed using a publicly available time-series dataset comprising renewable generation, load demand, electricity prices, battery state of charge (SoC), and supercapacitor SoC. Feature engineering, incorporating sinusoidal temporal representations and Min-Max normalisation, was applied to enhance learning stability and capture cyclical demand and generation patterns. The DQN agent was trained over 50 episodes and benchmarked against a conventional RBC strategy under identical operating conditions. Training performance demonstrated progressive policy improvement, with cumulative rewards increasing from approximately -1200 to -400, indicating enhanced decision-making capability during learning. The learned controller exhibited adaptive energy scheduling through dynamic utilisation of the supercapacitor and selective grid interaction in response to varying operating conditions, whereas the RBC followed a deterministic control strategy with limited flexibility. However, comparative evaluation revealed that the DQN did not consistently outperform the RBC in cumulative economic performance, suggesting the need for further refinement of the reward function, training process, and hyperparameter configuration. Nevertheless, the proposed framework demonstrates the feasibility of applying deep reinforcement learning to coordinated battery–supercapacitor energy management and highlights its potential to enhance operational flexibility and intelligent resource utilisation in renewable microgrids. The study contributes a dataset-driven reinforcement learning framework that provides a foundation for future research on advanced AI-based energy management systems and the integration of more sophisticated reinforcement learning algorithms for resilient and sustainable microgrid operation.