Mission-Horizon Hierarchical Reinforcement Learning for Fleet-Level PHM Under Resource-Constrained Operations

Abstract

Fleet-level prognostics and health management (PHM) requires decisions that extend beyond diagnosis, remaining useful life estimation, and maintenance timing. Mission commitment, aircraft dispatch, maintenance, and spare-parts support consume the same finite sustainment capacity but operate at different temporal scales. This study investigates whether explicitly separating a mission-horizon commitment decision from day-to-day sustainment execution improves fleet-level decision quality, and under which operating regimes such a hierarchy is valuable. We develop a mission-horizon hierarchical reinforcement learning (MH-HRL) framework in which a learned weekly policy selects a portfolio of missions from a seven-day forecast and specifies a coarse sustainment posture, while a fixed daily executor implements aircraft assignment, maintenance prioritization, and replenishment decisions. Fleet evolution is represented by a next-event discrete-event simulator with stochastic mission arrivals, degradation/failure processes, repairs, and logistics lead times. Five decision policies—reactive rule-based control, rolling-horizon planning, flat PPO, a hierarchy without the learned posture signal, and the full MH-HRL policy—are evaluated over nine mission-load/support-capacity regimes using 10 paired common-random-number seeds (450 formal runs). The hierarchical policies consistently improve the composite mission-readiness-cost utility, with the full policy outperforming the rule baseline in all nine regimes after Holm correction. The benefit is not obtained by maximizing mission acceptance: the hierarchical policy accepts fewer missions but achieves higher accepted-mission success and fleet availability while substantially reducing failure-driven sustainment cost. Ablation further shows that approximately 97% of the Smart vs. Rule improvement is attributable to the weekly daily temporal decomposition rather than the additional posture signal. These results support that the principal value of hierarchy in the studied fleet-PHM setting is long-horizon commitment control, and its marginal value increases with mission pressure.