Robust Industrial Manipulation Under Test-Time Perturbations in RLBench and Colosseum

Abstract

Robustness under test-time shift remains a major bottleneck for learning-based robot manipulation in cameraguided industrial workcells. Policies trained only in nominal simulation often lose performance when lighting, background appearance, table appearance, or object appearance changes at deployment. This paper studies this problem in a fixedcamera Franka Panda workcell using RLBench and Colosseum. We compare three training regimes under the same front, left-shoulder, and right-shoulder camera setup: a standard Robotic View Transformer (RVT) baseline, RVT with expanded targeted domain randomization (TDR), and RVT with expanded targeted domain randomization plus adaptive hard-example sampling (AHS). Expanded TDR broadens the training distribution over deployment-relevant Colosseum perturbation identifiers while keeping the backbone and sensor configuration fixed. AHS then updates replay weights online so that persistently low-scoring perturbation identifiers receive more attention without collapsing training onto only a few difficult cases. Across the four-task benchmark, expanded TDR improves the overall mean score from 32.4 to 50.0, and AHS further improves it to 58.0. The worst-ID score increases from 0.0 to 28.0, the Tail-2 score increases from 9.8 to 42.0, and the summary robustness-curve area increases from 14.6 to 44.0. The largest gains occur on the harder Close Box and Insert Onto Square Peg tasks. These results show that broadening the coverage of perturbation and adapting the replay toward the hard-tail is an effective and practical recipe for robust industrial manipulation.