Surrogate-Assisted Deep Reinforcement Learning for Commercial HVAC Optimal Control Considering Thermal Inertia

Abstract

Commercial buildings can provide operational flexibility through HVAC control because thermal inertia enables short-term load shifting under time-of-use (TOU) electricity pricing. However, direct reinforcement learning interaction with a high-fidelity building simulator is computationally expensive and may generate undesirable exploratory actions. This paper develops a surrogate-assisted Deep Q-Network (DQN) framework for commercial building HVAC control. EnergyPlus simulation data from a representative large office model configured for Nanjing, China, are used to train a black-box deep neural network surrogate that approximates short-term indoor temperature and HVAC power dynamics. The DQN agent is trained and evaluated through rollout in the validated surrogate environment, rather than through EnergyPlus closed-loop control. The control formulation uses price and occupancy look-ahead features, CORE_MID temperature, discrete cooling setpoint actions, and a reward considering operating cost, comfort, unnecessary cooling, action smoothness, and abnormal temperature variation. The surrogate predicts CORE_MID temperature with an RMSE of 0.260°C and an MAE of 0.191°C. Results show reduced HVAC energy, operating cost, and high-price-period energy use while maintaining office-hour thermal comfort.