An Automated Framework for Self-Adaptive Quality Assurance in Software Systems Using Policy-Based Reinforcement Learning
TL;DR
This paper presents a framework that removes both limits by using policy-based RL, and improves on a value-based baseline (DQN) with statistical significance and performs on par with a maximum-entropy method (SAC) under the tested settings, while keeping good sample efficiency and stability.
Abstract
Self-adaptive information systems must keep their quality requirements while their environment changes at run time. Building the adaptation logic by hand is difficult, because design-time uncertainty makes it impossible to foresee every environmental change. Online Reinforcement Learning (RL) can build this logic automatically. However, the value-based RL methods used so far have two practical limits: the exploration rate must be tuned by hand, and continuous states must be discretised by hand. This paper presents a framework that removes both limits by using policy-based RL. The Analyze and Plan phases of the MAPE-K loop are redefined as a single policy-based decision step, and Proximal Policy Optimization (PPO) is applied for online adaptation in continuous and discrete action spaces. The framework is evaluated on two systems: a self-adaptive web application and a predictive process-monitoring system. Across four workload patterns and two concept drifts, the framework learns effective policies without exploration tuning or state discretisation. It improves on a value-based baseline (DQN) with statistical significance and performs on par with a maximum-entropy method (SAC) under the tested settings, while keeping good sample efficiency and stability.