RoboTalk is introduced, a synthetic data-generation pipeline and dataset of 7,950 multimodal trajectories spanning 53 mobile-manipulation kitchen tasks for training small VLMs to communicate and coordinate and fine-tuning open-source models on this dataset can reach 77% success on novel held-out tasks.
Abstract
Multi-robot collaboration could enable more efficient and scalable solutions to complex robotic tasks, but collaboration under partial observability remains challenging. Natural-language communication offers a promising approach to coordinating robots under partial observability. However, in decentralized manipulation, jointly learning explicit inter-robot communication and skill-level action selection from multimodal demonstrations remains underexplored for small vision-language models (VLMs) intended for on-device deployment. To address this gap, we introduce RoboTalk, a synthetic data-generation pipeline and dataset of 7,950 multimodal trajectories spanning 53 mobile-manipulation kitchen tasks for training small VLMs to communicate and coordinate. The dataset includes a leader-follower planning protocol, tool calls (perception, manipulation, navigation, and communication), rationale traces, and diversified natural-language communication. Fine-tuning open-source models on our dataset can reach 77% success on novel held-out tasks, a significant improvement over the untuned open source models, which had a success rate of around ~2%.
MotorMind is introduced, a robot manipulation harness that connects VLM-proposed mid-level actions to deterministic robot control and feedback, with asynchronous monitoring and background memory updates, and shows that a general-purpose VLM, when equipped with an appropriate mid-level action representation and asynchro...
This work introduces DuoMind, a distributed hierarchical framework for multi-robot coordination through semantic communication, and develops RoboPoly, a benchmark comprising long-horizon manipulation tasks that require coordinated, closed-loop execution under distributed control.
Han-Chu Zhou, De-Chen Gao, Hang Wang et al.· 1 citation
A three-stage reinforced fine-tuning (RFT) pipeline for multi-agent VLAs is presented, which freezes the VLA and performs RL in its latent noise space and finds existing online RL for VLAs are less effective for hard multi-agent tasks, which are attributed to noisy co-exploration and unstable updates.
Rui-Xiao Xu, W. Kenny, Zhi-Qian Liu et al.· arXiv.org· 0 citations
This work presents RobotMover, a complete learning-based system for large-object manipulation that leverages human–object interaction demonstrations to train robot control policies and achieves strong performance in terms of capability, robustness, and controllability, outperforming both learned and teleoperation basel...
Tian-Yu Li, Joanne Truong, Tsung-Yen Yang et al.· IEEE Transactions on robotic...· 2 citations
This work presents AeroManip-VLA, a scalable benchmark for aerial VLA data generation and policy evaluation, and evaluates a range of imitation learning and VLA baselines across different task settings, revealing their performance characteristics and failure modes.
Rui Huang, Yan-Lin Mu, Li-Dong Li et al.· arXiv.org· 0 citations
DREAM is presented, a framework that generates fine-tuning data for a pretrained VLA from a captured workspace and a language instruction, without requiring a task-specific human demonstration, and whether it can serve as a scalable data-collection system for the deployment workspace.
Makoto Sato, T. Matsushima, Yutaka Matsuo et al.· 2 citations
MIT researchers developed FloatForm, a swarm of small aquatic robots that snap together like ants forming a raft, assembling into reconfigurable structures on the water.
MIT News · Artificial Intelligence· news.mit.eduJun 26, 2026
To help robots do chores in places like homes and factories, a new approach from MIT uses one language model to clarify users’ instructions, then another to ignore irrelevant info.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.