Controller Mimicking for On-the-Fly DES Reachability Control
TL;DR
CMRL is proposed, a Controller Mimicking Reinforcement Learning approach that trains a Deep Q-Network agent directly as a reachability controller on a pre-labeled monolithic plant by using a fast simulation environment derived from a solution, which completely decouples training from the costly on-the-fly synthesis procedure.
Abstract
One of the primary challenges in synthesizing controllers for Discrete Event Systems (DES) is the exponential growth of the composed state space. On-the-fly Directed Controller Synthesis (OTF-DCS) mitigates this by incrementally exploring the system, but its performance depends on the quality of the exploration heuristic. Recent reinforcement learning approaches learn effective heuristics but require repeatedly executing the expensive OTF-DCS algorithm during training, making the process computationally heavy. In this work, we propose CMRL, a Controller Mimicking Reinforcement Learning approach that trains a Deep Q-Network agent directly as a reachability controller on a pre-labeled monolithic plant. By using a fast simulation environment derived from a solution, CMRL completely decouples training from the costly on-the-fly synthesis procedure. This yields substantially lower training times (up to 81% reduction) while still delivering synthesis-time effectiveness that is on par with previous approaches.