Improving Long-Horizon Credit Assignment in Soft Actor-Critic for Bidirectional Utility Grid-Connected Microgrid Energy Management Systems

Abstract

The microgrid energy management system coordinates the operations of multiple microgrid components. Deep reinforcement learning algorithms, such as the Soft Actor-Critic (SAC) algorithm, are well known for off-policy learning, maximum entropy, and high sample efficiency, and SAC is used as a control agent in MG-EMS. However, the complex nature of MG-EMS and the additional complexity of utility grid connection with import and export pricing cause the SAC agent to have difficulty with long-term reward propagation due to weak long-horizon credit assignment, thereby making the agent inclined to learn near-term reward signals. The LHR-SAC method is proposed to address this issue by integrating a TD( $\lambda $ ) auxiliary critic loss with a multi-step target computed on sampled sequences using backward recursion with adaptive $\lambda $ . It enhances long-term reward propagation and long-horizon credit assignment in standard SAC. The method is evaluated in two different MG-EMS experimental setups using three metrics in each experiment across different forecast horizons, with accurate import and export price data and noisy forecast conditions. The LHR-SAC agent outperformed standard SAC and a prior PPO-based method across MG-EMS operations with import and export pricing. The LHR-SAC method is beneficial for MG-EMS tasks that require long-horizon credit assignment.