Surrogate Model Assisted Deep Reinforcement Learning for Integrated Energy Management of Smart Building

Abstract

Buildings are increasingly evolving from passive energy consumers into proactive energy hubs through the integration of on-site renewable generation, electrical storage systems (ESS), and flexible load management. This enables developing energy-efficient, low-carbon and cost-effective smart buildings. However, achieving the objective requires effective coordination of the heterogeneous energy resources in buildings to fully unlock the operational flexibility. This task is particularly challenging for commercial buildings where the heating, ventilation and air-conditioning (HVAC) system constitutes the dominant flexible load and shows substantial operational flexibility. The operation of the HVAC system is governed by multizone thermal dynamics that exhibit strong temporal and spatial couplings, making the joint optimization of HVAC operation and other flexible resources computationally challenging. To address this challenge, this paper proposes a surrogate model assisted deep reinforcement learning (DRL) framework for smart building integrated energy management. The framework incorporates a newly developed low-dimensional and physically meaningful surrogate model of multi-zone HVAC control into the integrated energy management. The surrogate model provides a compact and informative state representation, significantly reduces the dimensionality of state and action space, and enables a low-dimensional state-transition model for efficient policy learning of DRL agents. Case studies based on real-world data demonstrate that the proposed surrogate model assisted DRL significantly improves convergence rate compared with conventional DRL training on raw high-dimensional statespace model. Moreover, the results show that the substantial operational flexibility of multi-zone HVAC system can be effectively exploited to reduce building electricity cost.