HIGenNTO is presented, a framework that synthesizes humanoid-scene interaction motion references by optimizing the initial noise of a pretrained text-conditioned motion model under sparse spatiotemporal and scene constraints, establishing a scalable path from high-level task descriptions to physically executable humanoid interactions.
Abstract
Humanoid robots can acquire complex skills by imitating kinematic humanoid motion references, yet reliable references for contact-rich interactions remain difficult to obtain: motion capture deteriorates under occlusion and close physical contact, while retargeting introduces additional contact and geometric inconsistencies. We present HIGenNTO, a framework that synthesizes humanoid-scene interaction motion references by optimizing the initial noise of a pretrained text-conditioned motion model under sparse spatiotemporal and scene constraints. The same formulation satisfies desired contacts, avoids collisions, and maintains stable support while retaining the prior's realism and temporal coherence, generating interaction motions from scratch and composing long-horizon behaviors stage-wise. Across robot-environment and robot-object tasks, HIGenNTO produces motions that can be executed by tracking policies in simulation and used to train depth-conditioned visuomotor policies operating solely from onboard sensing. We deploy these policies on a Unitree G1 across four contact-rich tasks. Finally, the task specifications themselves can be written by a coding agent, which proposes interaction tasks and compiles them into prompt, constraint, and scene programs, authoring three of our eight evaluated tasks and four further behaviors. Together, these results establish a scalable path from high-level task descriptions to physically executable humanoid interactions.
Humanoid motion tracking provides a scalable interface for transferring kinematic human motion to whole-body robot control, but highly dynamic and contact-rich references remain difficult because underactuation and discontinuous contacts can restrict the closed-loop state distributions explored during training. Externa...
Chao-Jie Fu, Cheng-Kai Su, Lei Jiang et al.· Frontiers in Neurorobotics· 0 citations
PASSAGE, a perception-conditioned planner--tracker framework for humanoid traversal, using virtual reality and inertial motion capture to collect 100 h of scene-aligned human motion across 1,500 cluttered scenes and tests across 50 unseen physical layouts demonstrate traversal without prebuilt maps or offboard computat...
CHOREO, a framework for training-free composition of heterogeneous humanoid skills, demonstrates that executable trajectories provide a scalable interface for accumulating and composing pretrained humanoid capabilities.
Zi-Yi Sun, Jing-Wen Chen, Yu-Xin Wang et al.· arXiv.org· 1 citation· ⚡1
Captured human-object interactions provide rich supervision for humanoid loco-manipulation, but they are sparse, heterogeneous, and not directly executable by robots. We introduce InterMimicGen, a self-evolving motion-imitation framework in which robot motion data and a tracking policy improve each other. First, we con...
Yu-Chen Zhang, Sirui Xu, Jin-Hong Li et al.· 2 citations
Third-person human videos offer a scalable alternative to task-specific reward engineering, teleoperation, and manually scripted demonstrations for humanoid loco-manipulation. Recent video-to-humanoid systems have shown that noisy human motion, object trajectories, and contacts can be extracted from unstructured videos...
Partnered human-humanoid interaction couples locomotion with continuous physical contact. A humanoid needs to coordinate with a person's motion while responding to interaction forces and maintaining stable and natural movement. We present CoDance, a framework for learning reactive and compliant human-humanoid interacti...
From feet to fingertips — we are teaching robots intelligent whole-body control, fine dexterity, and teamwork to complete a broad range of complex tasks.