Highway Ramp Merging Behavior of Autonomous Vehicles Using Human-Inspired Structured LSTM with Reward Machine Reinforcement Learning

Abstract

For autonomous vehicles (AVs), safe and efficient highway on-ramp merging remains a challenging decision-making problem. To jointly improve safety, merging success, and driving efficiency, this paper proposes a novel human-inspired structured Long Short-Term Memory (HIS-LSTM) with reward machine reinforcement learning (RMRL). To ensure highway driving safety, an RMRL framework for on-ramp merging is first developed by incorporating the safe distance derived from the Vienna Convention into both the safety constraints of the Deep Q-Network (DQN) and the reward function. For the ramp merging task, HIS-LSTM is designed to emulate the cognitive process of human drivers. Specifically, it first monitors the motion patterns of surrounding vehicles (SVs), then analyzes their potential driving intentions, and subsequently fuses this behavioral understanding with the features of the ego vehicle (EV) to generate a reference speed. Simulation experiments are conducted in simple and complex highway on-ramp merging scenarios built in highway-env, with comparisons against eight baseline methods. Experimental results show that the proposed method achieves a superior trade-off among safety, merging success, and efficiency, while exhibiting strong robustness across different traffic complexities.