Trajectory-Based Residual Deep Reinforcement Learning for Electric Vehicle Charging Scheduling With Grid Stability Constraints

Abstract

Cooperative electric vehicle (EV) charging scheduling at an electric vehicle charging station (EVCS) must balance charging revenue, user service outcomes, and aggregate charging power. Threshold-triggered grid penalties introduce non-monotonic and non-additive reward characteristics that challenge conventional value-factorization methods. This article proposes a trajectory-based Residual Q-Network (RQN) for cooperative EV charging scheduling under centralized training and decentralized execution (CTDE). During centralized training, an auxiliary residual network extracts the mean and maximum statistics of individual Q-value trajectories and generates agent-specific correction factors before value aggregation. During decentralized execution, each charging pile selects its action using its local observation and individual action-value function. In the simulated EVCS environment, the proposed RQN achieves the highest mean total reward among seven representative value-decomposition, actor–critic, and policy-factorization baselines under the main setting. Ablation results demonstrate the contributions of residual correction and the two trajectory descriptors. Across different agent counts and three harder operating scenarios, RQN also retains the highest mean total reward among the eight compared methods, while the reward-preference analysis shows that the learned policy responds consistently to different scheduling priorities. Note to Practitioners—This article is motivated by the charging-scheduling problem faced by EVCS operators, who must coordinate stochastic charging demand, operating revenue, user service outcomes, and aggregate charging power. Threshold-triggered grid penalties make this coordination difficult because a charging decision that benefits one vehicle may reduce the common reward after the aggregate output crosses the penalty threshold. The proposed RQN addresses this issue through CTDE. During centralized training, trajectory-based residual correction improves the estimation of the coupled effects of individual charging decisions. During decentralized execution, each charging pile selects its charging action from its local observation and individual action-value function. The simulation results indicate that the proposed method can coordinate charging loads under different system scales, objective pReferences, and harder operating conditions. The present study focuses on algorithmic evaluation in a simulated EVCS environment. Application to an operating charging station would additionally require validation with field data and physical charging equipment while accounting for communication, measurement, and hardware constraints.