TL;DR
A hybrid reinforcement learning (RL) framework that jointly controls queue management and bandwidth allocation in bursty multi-service networks and demonstrates the effectiveness of coordinated learning-based control for stable and QoS-aware operation in bursty networked systems.
Adaptive queue management must balance throughput, delay, packet loss, and fairness under changing traffic and resource conditions. This paper proposes TS-D3QS, a traffic-state-aware queue scheduler that formulates multi-queue resource allocation as a discrete reinforcement-learning problem. The scheduler observes normalized queue and port features and selects one of 16 interpretable allocation profiles, each jointly specifying bandwidth shares, shared-buffer shares, and priority multipliers. A Dueling Double-DQN learner separates state value from action advantage and use a Double-DQN target to reduce value overestimation. Experiments in a reproducible four-queue simulator under bursty and non-stationary traffic show that TS-D3QS reduces latency by 7.54%, reduces aggregate loss by 3.21%, improves Jain fairness by 3.30%, and improves reward by 6.01% over vanilla DQN, while classical CoDel-like and WFQ-like rules remain competitive on selected objectives.
Reliable low-latency communication is a critical requirement in enterprise wireless networks such as hospitals, offices, and campuses. This paper proposes an earliest deadline first (EDF)-Lyapunov-Robbins-Monro (ELR), a stochastic scheduling algorithm for IEEE 802.11bn (Wi-Fi 8) Multi-Access Point Coordination Coordinated-Spatial Reuse (MAPC C-SR) networks that jointly accounts for queue stability and deadline-aware latency regulation under bursty traffic. A Lyapunov drift-based criterion for a group is adopted to ensure queues remain stable under varying traffic loads. Since the optimal balance between queue backlog and deadline urgency cannot be determined a priori under bursty traffic, EDF term is incorporated into the selection metric with a tunable balance parameter $\alpha$, governed by Robbins-Monro stochastic approximation scheme. The proposed algorithm addresses the inability of existing schedulers to track sudden congestion under bursty traffic, by dynamically adjusting $\alpha$ to suppress sharp delay spikes. Simulations over a four-access point (AP) enterprise deployment under bursty Markov-Modulated Poisson Process (MMPP) traffic demonstrate that ELR achieves 14.23%, 13.26%, and 7.97% reduction in 99th percentile delay over maximum number of packets (MNP), oldest packet (OP), and traffic alignment tracker (TAT) respectively under high load with 16 stations (STAs).
The rapid growth of multimedia streaming poses critical challenges, including bursty traffic and congestion, leading to playback delays. The existing separate prediction and control mechanisms for multimedia traffic scheduling, which are based on software-defined networks (SDN), are unable to proactively manage bursty traffic under uncertain conditions. This limitation is particularly evident in SDN-enabled backbone and multimedia-aware access networks, which typically assume centralized control and stable topologies. They lack integration of traffic prediction, traffic shaping, and real-time perception scheduling through reinforcement learning, resulting in low efficiency when exploring multiple paths in dynamic networks. To address this challenge, we propose PPO-MS (Proximal Policy Optimization-based Multimedia Scheduler), an SDN-based multimedia traffic scheduling algorithm integrating three key innovations: 1) A novel LSTM+HTB synergy where LSTM’s confidence intervals dynamically adjust HTB (Hierarchical Token Bucket) shaping parameters, enabling adaptive rate control under prediction uncertainty and overcoming the limitations of static LSTM+HTB hybrids; 2) A Deep Reinforcement Learning (DRL)-optimized path pruning method that reduces state and action spaces by generating a constrained set of $k$ disjoint candidate paths via an improved redundant tree algorithm. Unlike traditional multi-path schemes, this method tightly couples path preselection with the RL decision loop for adaptive, context-aware routing; 3) Generalized Advantage Estimation (GAE)-accelerated PPO for stable convergence in dynamic environments. In contrast to prior works (e.g., LSTM+RL for QoE or standalone tree algorithms), PPO-MS uniquely unifies these modules through confidence-aware traffic shaping and hierarchical decision-making, validated via comparative experiments. Results demonstrate that PPO-MS, through the synergistic integration of confidence-aware traffic shaping and DRL-optimized path pruning, significantly outperforms decoupled baselines. In particular, via isolation studies against simpler alternatives (e.g., mean-prediction and fixed-margin shaping), the confidence-aware shaping mechanism is validated to be superior under bursty traffic conditions. Overall, PPO-MS reduces end-to-end latency by 17.3% and packet loss by 32.4% while achieving 24.4% better load balancing during traffic bursts.
Time Sensitive Networking (TSN) is a key technology for deterministic communication in industrial control systems. Its Cyclic Queuing and Forwarding (CQF) mechanism can provide transmission guarantees with low latency and low jitter. However, existing studies often treat routing and scheduling as separate problems, overlooking their strong coupling relationship. Meanwhile, the injection of CQF with time slot offsets may cause some flows to exceed their deadlines and become unschedulable. This paper proposes a Genetic Algorithm-based Joint Routing and Scheduling Algorithm for CQF (Gene-JRSC). The algorithm aims to maximize the number of schedulable flows and establishes an Integer Linear Programming (ILP) model. A genetic algorithm (GA) is employed to solve the model, achieving joint optimization of routing strategy and schedulability. Simulation experiments demonstrate that, compared to existing algorithms, Gene-JRSC significantly improves the scheduling success rate and queue resource utilization under different network scales and traffic loads.
Dynamic wireless resource allocation in multi-cell networks is challenging due to non-stationary traffic, intercell interference coupling, and heterogeneous quality-of-service (QoS) constraints. Conventional schedulers and standalone metaheuristics lack adaptability across operating regimes, while deep reinforcement learning (DRL) methods often incur high training complexity and stability limitations. This paper proposes a context-aware reinforcement hyper-heuristic framework for dynamic wireless resource allocation. A contextual bandit controller hierarchically selects among multiple low-level optimization heuristics based on real-time network state features. A multi-objective reward design jointly optimizes throughput, fairness, power efficiency, and allocation stability. We establish sublinear regret guarantees under the contextual bandit model and prove convergence under standard stochastic approximation conditions. Extensive simulations over 5,000 large-scale multi-cell instances demonstrate consistent improvements over proportional fair scheduling, evolutionary methods, and DRL-based allocators in throughput, Jain's fairness index, convergence speed, and robustness to traffic perturbations. Statistical tests confirm the significance of the gains. The results indicate that reinforcementdriven hyper-heuristic orchestration provides a scalable and theoretically grounded solution for dynamic wireless resource management.
Achieving deterministic latency for time-sensitive flows within integrated 5G and Time-Sensitive Networking (TSN) ecosystem requires the active mitigation of stochastic delays inherent in 5G New Radio (NR). While existing research typically relies on pessimistic guard bands or over-provisioned time-domain resources via wired TSN mechanisms, these approaches fail to adaptively reserve NR resources under dynamic channel conditions to suppress Packet Delay Variation (PDV). This work addresses this gap by proposing a joint NR MAC scheduling and Link Adaptation (LA) framework. We introduce Link Adaptive Semi-Persistent Scheduling (LA-SPS), a framework that ensures cycle-synchronous uplink opportunities by dynamically reconfiguring resource budgets and modulation parameters from real-time channel feedback. To manage the combinatorial complexity of joint resource allocation, we employ a Graph Neural Network (GNN) to encode scalable network states and Proximal Policy Optimization (PPO) for stable, real-time decision-making. This modular framework functions as a radio-side control loop designed for seamless coupling with end-to-end Time-Aware Shaper (TAS) scheduler, enabling a fully co-adaptive industrial network.