Microgrid energy management optimization using PPO with curriculum learning trained in virtual simulation environment

Abstract

Microgrid energy management systems (EMS) require real-time, economically optimal, and safe dispatch strategies under high renewable uncertainty. While Deep Reinforcement Learning (DRL) offers promising online decision-making capabilities, standard DRL algorithms struggle with the “cold start” problem, slow convergence, and severe Sim-to-Real performance degradation when transferred from simulation to real-world operating conditions. To address these challenges, this paper proposes a novel Curriculum Learning-enhanced Proximal Policy Optimization (CL-PPO) framework for grid-connected microgrid energy management. By structuring a high-fidelity virtual environment into a three-stage progressive curriculum (Basic, Volatile, and Extreme), the agent systematically evolves from learning fundamental arbitrage rules to mastering robust dispatch under severe boundary conditions. Experimental results demonstrate that curriculum-guided knowledge transfer reduces the required convergence steps by approximately 38% compared with Baseline PPO. In typical-scenario evaluations, CL-PPO reduces daily operating costs by 7.1% relative to Baseline PPO and achieves a near-optimal 2.4% economic gap relative to the non-causal Mixed-Integer Linear Programming (MILP) oracle, requiring only 12.5 milliseconds per inference. Furthermore, an offline Sim-to-Real replay evaluation was conducted using approximately 90,000 previously unseen operational records from a real-world campus microgrid. In a replay environment parameterized according to the target microgrid’s energy-storage-system characteristics, CL-PPO demonstrated strong cross-domain robustness against measurement noise and operational disturbances, achieving an average total constraint violation rate of 1.2%. No policy-generated command was applied to the physical microgrid during this evaluation. Ultimately, this research enhances the interpretability of DRL policy evolution and provides a safety-aware and computationally efficient training framework with promising potential for future physical deployment, subject to further hardware-in-the-loop and on-site closed-loop validation. A curriculum learning-enhanced PPO (CL-PPO) framework is proposed for safety-aware and computationally efficient energy management in grid-connected microgrids. A dual-condition transition criterion jointly evaluates reward and safety violation rate to guide curriculum progression. A decoupled reward function separates economic incentives from safety-boundary penalties, improving policy interpretability. CL-PPO reduces the required convergence steps by approximately 38% and narrows the economic gap relative to the MILP oracle to only 2.4%. Offline Sim-to-Real replay evaluation using approximately 90,000 unseen real-world operational records demonstrates strong cross-domain robustness, with an average total constraint violation rate of 1.2%.