A Bilevel Reinforcement Learning Framework for Coordinated EV Charging and Dynamic Pricing

Abstract

This paper presents a bilevel Reinforcement Learning (RL) framework for optimizing Electric Vehicle (EV) charging through price-mediated coordination between grid operators and charging stations. Unlike prior work relying on direct control or manual subgoal engineering, the proposed approach uses dynamic pricing as an implicit coordination signal to address a complex multi-objective optimization problem involving grid stability, user satisfaction, and economic efficiency. To manage this complexity, the problem is decomposed into two levels comprised of an upper-level Distribution System Operator (DSO) that determines dynamic pricing strategies, and multiple lower-level Load Aggregators (LAs) responsible for EV charging decisions at individual stations in response to these prices. This bilevel structure captures the leader–follower interaction between DSOs and LAs, with each level operating at different temporal scales. Deep Deterministic Policy Gradient (DDPG) agents are deployed at both levels, enabling adaptive decision-making under operational constraints. Extensive simulations compare the framework against multiple Rule-Based Control (RBC) baselines. Results demonstrate that the DDPG-based DSO achieves a 42.4% higher mean reward and 19.1% higher profit compared to the best-performing RBC baseline, while preserving grid stability and user satisfaction. These results validate the effectiveness of bilevel RL for complex energy optimization problems, highlighting its potential as a scalable control paradigm for smart management systems.