Automatic Operation Algorithm for Stormwater Pumping Stations Based on Heuristic-Guided Reinforcement Learning

Abstract

Automated operation of urban stormwater pumping stations is a sequential decision-making problem that must prevent detention-basin overflow during intense rainfall while coordinating pump switching, pump use, and dry-running risk. This study proposes a heuristic-guided reinforcement-learning framework in which a genetic algorithm (GA) or particle swarm optimization (PSO) proposes feasible pump combinations during the online training of a Double Deep Q-Network (DDQN). At each decision time the heuristic searches for a combination that reflects four operational criteria. At each training step, the heuristic proposes the action with a probability that decays from 0.6 to 0 over training, and the resulting transition is added to the replay buffer online, reducing uninformative random exploration in early training. As training proceeds, the probability of selecting heuristic actions is gradually decreased, thereby transferring the knowledge obtained from search to a reactive neural-network policy. The proposed method was compared with a conventional DDQN and heuristic-only policies. Evaluation using 6696 rainfall scenarios for the Gasan Stormwater Pumping Station in Seoul showed that, under an identical reward setting, GA-guided DDQN reduced the loss faster than Regular DDQN. In a limited-data experiment restricted to the first 160 training scenarios, the mean test reward did not differ meaningfully between GA-guided DDQN and Regular DDQN (152.69 ± 1.14 versus 153.17 ± 0.56 across three seeds). With the full training set and five random seeds, GA-guided DDQN attained the lowest maximum water level (6.03 ± 0.09 m versus 6.22 ± 0.08 m for Regular DDQN) but a higher dry-running risk proxy (4.87 ± 4.39 versus 3.30 ± 2.75), whereas PSO-guided DDQN reduced both pump on/off changes (32.54 ± 5.67 versus 39.30 ± 6.41, −17.2%) and the dry-running risk proxy (2.16 ± 1.09 versus 3.30 ± 2.75, −34.7%). The heuristic-only policies attained the lowest dry-running risk proxy but the highest maximum water level and the largest number of pump on/off changes among the five models. No overflow occurred in any of the 2700 test scenarios for any of the five models. The preferred model therefore depends on the operating objective: GA-guided DDQN for the lowest peak water level, and PSO-guided DDQN for the fewest pump on/off changes and the lowest dry-running risk proxy. These results show that heuristic guidance can shape early exploration without expert operating data and transfer the resulting knowledge to a policy that acts without a per-decision search.