Dynamic Grid World Navigation using Q-Learning with Moving Obstacles and Terrain-Based Rewards

Abstract

This paper describes the concept, construction, and experimental assessment of a reinforcement learning (RL) tool for operating in a moving-obstacle and heterogeneous-terrain grid-world environment. The modeled system utilizes tabular Q-Learning, a model-free, off-policy temporal-difference (TD) learning algorithm to teach an autonomous robot the best navigation policy by engaging repeatedly with a stochastic, partially adversarial environment. In contrast to classical grid-world tests where threats are fixed and rewards are constant, the environment created in this research features two major complications: (1) obstacles that move using both random and pattern-based motion modes, thus resulting non-stationarity in the agent’s effective state transitions, and (2) different terrain types that provide different reward signals including normal tiles with a small negative step penalty, rough terrain with a larger traversal cost, and bonus tiles resulting in a positive incentive. The methodical results show a steady increase of total reward per episode, a decrease of mean episode duration, and a stabilization of the Q-value function after 300-400 training episodes. Besides presenting the main method, the authors also provide a comparative analysis against SARSA, Deep Q-Network (DQN), policy gradient methods, and baseline heuristics, corroborating that tabular Q-Learning is a strong performer in terms of solution quality at the considered problem scale and it also provides interpretable results. The article also sketches some theoretical limitations of the tabular approach in terms of scalability and describes possible future directions like function approximation, curriculum learning, and multi-agent configurations, etc.