Reinforcement Learning for Optimized Well Placement in Underground Hydrogen Storage

Abstract

Abstract Well placement design for underground hydrogen storage (UHS) faces the challenge of balancing hydrogen recovery, costs, and operational constraints. This study develops a rule-guided reinforcement learning (RL) framework to jointly optimize well count and well locations for a depleted gas reservoir in Alberta, Canada. The objective is long-term net present value (NPV), which is normalized as total reward. The problem is formulated as a constructive Markov decision process (MDP) in which a Soft Actor-Critic (SAC) agent sequentially builds a static well pattern. A rule-based feasibility layer removes inactive cells that violate reservoir boundary and minimum spacing requirements, while a STOP action allows the agent to terminate well placement when additional wells are no longer economically beneficial. Completed patterns are evaluated using a reservoir simulator over ten injection-withdrawal cycles. During training, the agent initially favored patterns near the maximum well limit to increase recovery, but gradually converged toward approximately 50 wells. This reduced recovery slightly while substantially improving NPV, indicating that the agent learned the trade-off between incremental recovery and the diminishing economic value of additional wells. The trained policy was then applied directly to two unseen geostatistical models. In Test Model 1, SAC selected 53 wells and achieved a recovery factor of 88.3% and an NPV of CAD 95.2 million. The 50-well geology-guided benchmark achieved 86.4% and CAD 65.1 million. In Test-Model-2, SAC selected 52 wells and achieved 88.2% and CAD 95.7 million, compared with 86.9% and CAD 75.6 million for the 50-well benchmark. These results correspond to NPV improvements of 46.3% and 26.7%. Results show that the recovery advantage is concentrated in the early cycles, whereas the economic benefit persists after the recovery curves largely converge. The optimized well patterns also contain feasible wells in geologically moderate locations, suggesting that the learned policy captures pattern-level interactions beyond the independent ranking of individual locations.