Optimizing production planning and control: state space design in reinforcement learning

Abstract

Modern production systems are increasingly challenged by volatile and complex market conditions. To maintain their competitiveness, the application of Reinforcement Learning (RL) in production planning and control has shown considerable promise. However, the state space in RL approaches is often excessively large and complex, which triggers the curse of dimensionality and hinders learning efficiency. A significant research gap exists regarding systematic methodologies for defining, evaluating, and iteratively optimizing the state space to improve both the transparency and learning performance. This paper addresses this gap by proposing a novel methodology consisting of two steps. First, an initial state space is systematically derived from corporate planning goals with the help of a feature map. Second, Shapley Additive Explanations (SHAP) values are utilized to quantify the influence of each state feature on the RL agent’s action selection and to iteratively improve the state space accordingly. The proposed methodology is validated using a real-world industrial use case from the machinery industry. The results indicate that the methodology successfully identifies dominant state features, leading to an improved learning behavior and enhanced transparency in the RL agent’s decision-making process.