Mitigating Covariate Shift in Offline Imitation Learning via Trajectory Synthesis

Abstract

Offline imitation learning (IL) is a practical route to embodied policy learning when online exploration is unsafe, expensive, or unavailable. Its main limitation is the closed-loop covariate shift: a policy trained only on fixed demonstrations can enter states outside the expert support, where behavioral cloning provides no recovery supervision and small errors compound. We propose Synthetic Imitation Learning (SynIL), an offline IL framework that augments demonstrations with synthetic recovery trajectories rather than only filtering or reweighting logged data. SynIL uses generative dynamics models to enlarge the training distribution around failure-prone states and retains transitions whose successor states move back toward the expert state manifold. In low-dimensional benchmarks, SynIL trains a lightweight latent world model to imagine short-horizon rollouts from suboptimal states and uses a positive-unlabeled state discriminator to assign proxy rewards without access to task rewards. In high-dimensional robot manipulation, SynIL instantiates the same principle with a training-free visual synthesis pipeline that combines pretrained video generation, segmentation, point tracking, and monocular depth estimation to recover image-action trajectories. Experiments on 15 D4RL tasks show that SynIL achieves first- or second-place performance on 8 tasks across navigation, locomotion, and manipulation. Real-world UR5 experiments improve the average success rate by 17.5% over behavior cloning, indicating that synthetic recovery data can improve robustness without online environment interaction. Note to Practitioners—This paper addresses a common deployment setting in embodied robot learning: a practitioner has a small set of successful demonstrations and a larger set of imperfect logs, failed attempts, and/or human corrections. We use expert or perfect demonstrations to denote successful task executions; the key issue is not the label itself, but the scarcity of such reliable recovery supervision. The practical message is that imperfect trials should not be treated as useless data. They often contain precisely the off-manifold states that a deployed policy will encounter after making a small mistake. SynIL turns these states into training signal by synthesizing short trajectories that move back toward successful behavior. In low-dimensional tasks, this is achieved with a learned latent world model; in visual manipulation, it is achieved with pretrained video generation and geometric post-processing. Since all synthesis is performed offline, the deployed policy has the same inference-time cost as the base imitation learner. For industrial manipulation, warehouse automation, laboratory robotics, and other safety-sensitive domains, SynIL reduces the need to collect many additional expert demonstrations. The approach is most useful when imperfect data contain meaningful partial attempts rather than arbitrary noise, and when generated trajectories can be anchored to real observations or filtered by an expert-aligned quality signal.