Adaptive Filtering with Reinforcement Learning against Model Mismatch in Low-Observable Settings

Abstract

State estimation of dynamical systems is a central problem in many applications. Optimal solutions for such problems are well established when the system dynamics are completely known (i.e., Kalman filter in the linear–Gaussian setting). When the assumed models are mismatched from the true system's transition, subspace identification methods are commonly employed but only under the assumption of fully observable systems. In this paper, we investigate linear time invariant (LTI) systems under a model mismatch and low observability setting, a scenario often encountered in practice. We propose a novel framework, dubbed reinforcement learning for adapting transition model of Kalman filter (RLTKF), in which an agent learns to select the state transition matrix used by a Kalman filter. We first show that the filter's estimation error can be bounded with respect to an observability metric and the model mismatch. Building on this analysis, we introduce an RLTKF agent incorporating observability knowledge in shaping the RL state. Regarding reward shaping, we investigate the ideas of optimizing the estimation error either directly or via reducing the model mismatch. We conduct numerical experiments to evaluate our RLTKF variants under different observability conditions. We show that RLTKF demonstrates significant improvements in estimation accuracy and robustness compared to classical methods. We also observe that minimizing model mismatch can be sub-optimal. Finally, we highlight the influence of observability information on RLTKF's performance.