Hierarchical Reinforcement Learning for Multi-Robot Task Allocation and Path Planning in Digital Twin Enabled Smart Warehouses

Abstract

As more companies embrace the use of autonomous mobile robots to organize their smart warehouses, one of the major issues facing them is the coordination of these fleets under very restrictive throughputs. Task allocation and path planning go hand in hand: if tasks are assigned without considering congestion, then each task can result in a conflict or a deadlock. Conversely, if path planning takes place totally reactively, then the total travel effort and energy lead to wasteful results. This research introduces TwinSync-HRL, a framework that involves the coupling of reinforcement learning for task allocation in multi-robot settings with classical search-based path planning. The allocation is based on a learned Q-learning policy to manage the assignment of pickup-and-delivery tasks to robots while the actual path planning takes place using A* with space-time reservation. TwinSync uses a digital twin of the overall warehouse in order to facilitate synchronization and to quantify the conditions under which this synchronization takes place. The results show that the learned Q-learning allocator is superior to both the greedy and auction-based methods in the sense that it clears all 20 tasks per episode without conflict and at the lowest possible distance (483 ± 5.3 cells traveled). However, it does have a slower rate of task clearing (under mean conditions of 100.6 ± 6.5 steps as compared to 84.4 to 86.5 for the other two methods). When the total synchronization latency time increases from 0 to 15 steps, the number of replans per episode goes from 17.4 to 37.9 and the total conflicts in execution triples.