CL4AD is presented, the first integration of curriculum learning into batched autonomous driving simulators by framing scenario selection as an unsupervised environment design problem, and utility functions that shape curricula based on success rates and the realism of the agent's behavior are introduced, in addition to existing regret-estimation functions.
Abstract
Batched simulators for autonomous driving have recently enabled training reinforcement learning (RL) agents at scale, encompassing thousands of traffic scenarios and billions of interactions within a matter of days. Although such high-throughput feeds RL algorithms faster than ever, their sample-efficiency has not kept pace: As the standard training scheme, domain randomization uniformly samples scenarios, thereby consuming a vast number of interactions on cases that contribute little to learning. Curriculum learning offers a remedy by adaptively prioritizing scenarios that matter most to policy improvement. We present CL4AD, the first integration of curriculum learning into batched autonomous driving simulators by framing scenario selection as an unsupervised environment design problem. We introduce utility functions that shape curricula based on success rates and the realism of the agent's behavior, in addition to existing regret-estimation functions. Large-scale experiments in GPUDRIVE demonstrate that curriculum learning achieves a 99% success rate a billion steps earlier than domain randomization, reducing wall-clock time by 77%, and outperforms heuristic curricula with static and dynamic attributes, with only one exception at the largest scale. An ablation under limited compute shows that curriculum learning improves sample efficiency by 67%. We also investigate how utility functions behave at scale, and how prioritized scenarios evolve during training. We release an implementation of CLForAD in GPUDRIVE.
Reinforcement learning (RL) has become an increasingly important framework for autonomous driving, while the CARLA simulator has emerged as a leading benchmark environment for training and evaluating RL-based driving agents. Despite rapid growth in this area, the literature remains fragmented, making it difficult to identify prevailing methods, experimental practices, and open challenges. This paper presents a comprehensive review of approximately 100 peer-reviewed studies that apply RL in the CARLA simulator. The surveyed works are organized into major methodological categories, including model-free, model-based, hierarchical, hybrid, and other specialized RL approaches. Our analysis shows that model-free RL overwhelmingly dominates the field, accounting for more than 80% of existing studies, with DQN, PPO, and SAC being the most frequently adopted algorithms. We also examine how these studies formulate driving problems through different state representations, action spaces, and reward designs, and we summarize the evaluation landscape in terms of metrics, towns, scenarios, and traffic configurations. Finally, we highlight persistent research challenges such as sparse reward design, generalization, sim-to-real transfer, safety, and limited behavioral diversity, and we discuss emerging directions that may help address these limitations. This review provides a structured reference for researchers entering the field and offers a foundation for future advances in RL-based autonomous driving in CARLA.
Reinforcement learning (RL) is being studied for autonomous driving (AD), but its value depends on the role it plays in a task, the action interface, the evaluation protocol, and the evidence from deployment. This survey examines RL-based AD in modular and end-to-end pipelines and relates reported methods to task formulation and deployment evidence. It maps safe RL, offline RL, model-based RL, and PPO/GRPO-style fine-tuning to maneuver selection, continuous control, world modeling, and VLM/VLA-based driving. It also reviews simulators, datasets, RL platforms, and VLA benchmarks, with attention to reward design, observation space, traffic complexity, and open-loop versus closed-loop evaluation. The survey then examines deployment barriers, including safety, Sim2Real generalization, data efficiency, computation, embodied alignment, and evaluation readiness. RL and VLM/VLA-based methods have shown promise, but current evidence is insufficient to support reliable real-world deployment: many reported results come from restricted scenarios and depend on engineered rewards or simulator assumptions.
Bin Shuai, Min Hua, Le-Tian Tao et al.· Communications in Transporta...· 0 citations
Robust navigation policies for autonomous agents must generalize across continuously varying environmental conditions such as turn rates, obstacles, friction, pits, and slopes. Curriculum generation provides a principled mechanism for improving generalization by progressively adapting training environments, but designing such curricula in a sample-efficient and automated manner remains challenging. This paper proposes a reparameterized curriculum generation framework for structured continuous environment parameters using unidirectional gradient-based optimization. To improve robustness in multimodal observation spaces consisting of image-based and scalar inputs, a distribution-shift regularization objective is incorporated to encourage the learning of finer-grained latent representations. The proposed method is evaluated across two continuous-control OpenAI Gym environments: a 2D obstacle-based Car Racing variant and Bipedal Walker variant, where coupled environment parameters jointly influence policy performance. Across five random seeds, our method consistently outperforms vanilla policy training, random parameter sampling, manual curricula, frontier-based methods, Self-Paced Reinforcement Learning (SPRL), Absolute Learning Progress with Gaussian Mixture Models (ALP-GMM), and reverse curriculum learning baselines. Ablation studies further demonstrate the effectiveness of the reparameterized curriculum mechanism across both environments, while highlighting environment-dependent benefits of the auxiliary regularization objective.
Behavioral cloning trains a policy offline on expert demonstrations, but deployment is closed loop: each action affects the observations the policy receives next. We study how much closed-loop driving competence a compact multimodal policy can acquire from offline demonstrations in the CARLA simulator. The policy uses five-frame histories of RGB images, LiDAR, vehicle telemetry, and lane waypoints to predict throttle, brake, and steering at 20 Hz. Demonstrations were collected in three stages, ending with a systematic route-generation procedure that enumerates spawn points and feasible maneuvers and verifies completed autopilot routes. The released 1.36 million parameter policy was trained on 236,882 windows, representing about 3.3 hours of driving from 448 captures. The resulting policy drives autonomously for hours on training and held-out routes. In our runs, it did so without collisions and also transferred qualitatively to an unseen CARLA town with different road geometry. We also observed recovery from large trajectory deviations, although we do not claim systematic recovery without controlled evaluation. We report offline metrics and distinguish measured results from qualitative closed-loop observations. We release the code, trained checkpoint, ONNX model, data sample, and an evidence audit for the reported claims.
This work uses Sample-based Model Predictive Control entirely in simulation as an automated, rapidly tunable expert to generate massive offline datasets and validate the robustness of this sim-to-real framework by successfully deploying complex loco-manipulation skills across different morphologies.
Martin Schuck, Maks Sorokin, S. Manni et al.· 0 citations
River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization is proposed, which achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks.
Yi-Fan Yao, Bo Pang, Xuan-Phi Nguyen et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
