LyMARL: A Lyapunov-Guided Multiagent Reinforcement Learning Framework for Energy-Aware Radio Resource Management

Abstract

Energy-efficient control in dense wireless networks requires jointly optimizing user association, scheduling, and base station (BS) activation under finite energy budgets. Existing Lyapunov-based approaches enforce energy constraints only asymptotically. Under finite-horizon operation, the myopic per-slot drift-plus-penalty (DPP) objective causes aggressive early BS activation, rapidly exhausting the budget and forcing prolonged service outages. Existing multiagent reinforcement learning (MARL) approaches rely on heuristic reward shaping or Lagrangian penalties without explicitly tracking cumulative violations over finite horizons. To address these limitations, we propose LyMARL, a Lyapunov-guided heterogeneous MARL framework that embeds DPP virtual queue dynamics into both the state representation and reward design of a centralized training with decentralized execution (CTDE) architecture. By jointly learning user and BS agents with inherently conflicting objectives, LyMARL enables trajectory-aware decision making that proactively distributes the activation budget over the finite horizon. Simulation results demonstrate that LyMARL outperforms Lyapunov-based, conventional MARL, and constrained MARL baselines in throughput and fairness, while sustaining near-seamless service continuity and stable BS activation throughout the finite horizon, avoiding the early budget exhaustion that plagues existing approaches. The learned policy generalizes well across varying user densities without retraining, confirming its scalability for energy-aware control in dense small-cell networks.