The findings suggest that hard carbon budgets can be controlled in expectation online at modest cost—a step toward operator-facing low-carbon logistics whose average emissions honour a binding carbon budget, though external validation on operational data and a risk-sensitive formulation that upgrades this average control into per-day compliance are still required before deployment.
Abstract
Urban last-mile delivery is a rapidly growing source of city-centre emissions, and decarbonizing it without eroding service quality has become imperative for climate goals. Operators are turning to multimodal systems that integrate road vehicles, off-peak metro freight, and electric drones—yet the optimal delivery channel varies dynamically with location and time. Current RL-based schedulers handle constraints via manually tuned penalty weights, lacking formal safety guarantees, and the feasibility of online carbon-cap enforcement under partial observability remains an open question. To address this, we model the problem as a Constrained Markov Decision Process (CMDP) and propose a demonstration-regularized Lagrangian deep RL algorithm. Our approach learns an online policy that is model-free at deployment—it controls emissions in expectation against a hard carbon budget, makes per-order decisions using only state observations, and operates without an emission model at test time (the demonstrator used at training time does access the emissions model, so “model-free” refers strictly to the deployment phase). Experiments on synthetic benchmarks and a Nanjing-inspired scenario—grounded in real metro topology and population-weighted demand—show that our policy achieves emissions within 1.3% of the offline optimum. It robustly tracks a ±17% carbon-budget band across a threefold daily volume range and a threefold city-scale range, with zero per-instance tuning. By contrast, a standard PPO with fixed penalty weights consistently degrades to single-mode selection. Our findings suggest that hard carbon budgets can be controlled in expectation online at modest cost—a step toward operator-facing low-carbon logistics whose average emissions honour a binding carbon budget, though external validation on operational data and a risk-sensitive formulation that upgrades this average control into per-day compliance are still required before deployment.
A hybrid optimization framework that combines greedy initialization with reinforcement learning to efficiently explore the charging station deployment problem is proposed and demonstrates stable performance across three evaluated deployment scenarios, indicating its potential applicability to increasingly complex charging infrastructure planning problems.
Smart freight corridors offer a practical pathway for connected and automated vehicle (CAV) deployment in freight transportation, but physical experimentation is expensive and existing approaches rely on predefined control policies that cannot capture adaptive behaviors. This paper presents an agent-based modeling (ABM) framework coupling a physical infrastructure layer, a connectivity layer (V2X), and a decision layer integrating reinforcement learning (RL) and multi-agent reinforcement learning (MARL) for platoon formation and charging coordination. We evaluate three scenarios (Baseline, Assisted, and Cognitive) using throughput, congestion, energy, emissions, and robustness metrics. Preliminary results indicate that the Cognitive scenario achieves higher throughput and lower congestion than the baseline, while the Assisted scenario delivers meaningful energy savings per kilometer through platooning. Sensitivity analysis shows that the throughput advantage of the smart corridor widens under conditions with high demand and that MARL coordination extracts greater utilization from fixed charging capacity than rule-based assignment.
Madelaine Martinez-Ferguson, Chun Wang, M. C. Camur et al.· 0 citations
The findings show that a single controller trained with surrogate safety indicators as learning objectives can improve operational performance while reducing safety-critical instability in work zones.
Israel Afriyie, Kwadwo Amankwah-Nkyi, Percy Agyei-Essiful et al.· Future Transportation· 1 citation
Advanced metering infrastructure (AMI) is the sensing backbone of the smart grid, and its reliability underpins urban energy services such as state estimation, demand response, and distributed-energy integration. When AMI uses cellular spectrum leased through a cognitive mobile virtual network operator (C-MVNO), allocating channels to data aggregation points (DAPs) each frame is difficult because three uncertainties interact: imperfect spectrum sensing, time-varying and cross-channel-correlated primary-user activity, and stochastic urban propagation. Classical Hungarian assignment is optimal per frame but blind to primary-user dynamics, while cognitive-radio heuristics ignore queue state and cross-channel structure. We propose a two-timescale hierarchy that couples these established tools in a new way: a Proximal Policy Optimization (PPO) agent decides, once per epoch, which opportunistic channels to expose, and an exact Hungarian solver performs the per-frame DAP-to-channel assignment. To our knowledge this is the first coupling of a learned cognitive layer with exact Hungarian assignment for cognitive-radio resource allocation. On a 3GPP TR 38.901-compliant simulator, PPO significantly outperforms a Bayesian-belief baseline and the Hungarian-only configuration in delivery ratio, latency, and a strict per-meter satisfaction metric, and is robust across independent seeds and sensitivity sweeps. An architectural ablation shows the DAP tier is a precondition for viability, not merely an optimization.
M. Al-Ali, Esteban Inga, Juan Inga et al.· Smart Cities· 0 citations
Large-scale public events generate concentrated passenger demand during egress periods, often overwhelming urban transit systems. This paper proposes a multi-modal evacuation framework that coordinates in-service buses temporarily diverted from existing lines and dedicated shuttle vehicles pre-positioned at depots. The problem is formulated as a two-layer stochastic optimization under travel time uncertainty: the upper layer determines pre-event shuttle fleet sizing, while the lower layer makes real-time dispatching decisions for both modes. We propose an Uncertainty-Aware Reinforcement Learning framework with Categorical DQN (UARL-CD) that learns a robust dispatching policy through a reward function aligned with the lower-level objective, explicitly accounting for travel time uncertainty via distributional value representation and stochastic training, with an action masking mechanism enforcing operational constraints. Simulation experiments based on a realistic stadium evacuation scenario demonstrate that the proposed framework significantly outperforms deterministic optimization and rule-based strategies, achieving a 31.6% reduction in evacuation completion time and a 48.4% reduction in average passenger waiting time compared to shuttles alone, while maintaining robustness to travel time uncertainty with only 4.0% performance degradation and online decisions executed within the 2-min decision interval.
HVAC control trades energy against thermal comfort, complicated by two building features: thermal mass spreads a setpoint change over hours, and the input-to-outcome mapping shifts across the year. Model-free algorithms such as PPO, SAC, and TD3 carry no model of building dynamics and cannot evaluate a setpoint’s downstream effect. We apply a Latent Dynamics Learning and Planning (LDLP) framework that learns a latent model of the building’s thermal response and plans with Monte Carlo Tree Search inside it. We also gate a reward formulation common in prior work, applying its comfort penalty only while the building is occupied. On Sinergym’s 5Zone environment under hot, mixed, and cool climates, LDLP is evaluated against PPO, SAC, TD3 and Sampled EfficientZero, an independent model-based controller run at the same budget. With energy normalized for the comfort achieved, LDLP consumes 4% to 26% less than PPO and SAC under the standard reward and 1% to 19% less under the gated reward. Running the same model with one simulation per decision, which removes planning, multiplies its normalized energy fivefold. Under the gated reward the deterministic TD3 policy degenerates onto a few fixed setpoints, so we report action diversity alongside the conventional metrics.
