SHRIMP is introduced, a system that allows users to automatically generate a hierarchical robot primitive plan using natural language and iteratively revise their plan through re-prompting and explicit correction, and once satisfied, execute it on the physical robot.
Abstract
As collaborative robots have entered domains such as manufacturing, agriculture, and healthcare, programming or adapting robot behavior typically requires robotic expertise that most end users lack. Natural language lowers this barrier. Recent advancements in large language models (LLMs) have made it feasible to translate natural language into robot task plans. However, language-based task specification suffers from semantic ambiguity, and generative models lack transparency for how language instructions become robot actions, making it difficult for users to validate the plan before execution. To address these issues, we introduce SHRIMP, a system that allows users to automatically generate a hierarchical robot primitive plan using natural language and iteratively revise their plan through re-prompting and explicit correction. At each revision, SHRIMP allows users to validate their plan in simulation, and once satisfied, execute it on the physical robot. Through a user study involving participants planning tabletop kitchen tasks (n=35), we validate that SHRIMP improves perceived control and enhances robot transparency. System videos and source code are available at https://wisc-hci.github.io/SHRIMP.
This work introduces RobotWorld, a challenging simulation testbed for robot use: turning instructions and observations into physical task execution through robot interfaces, and analyses task outcomes alongside execution traces, identifying both the capabilities that transfer and the gaps that prevent reliable completi...
Zhiqin Yang, Chen-Xin Li, Xiao-Meng Hu et al.· 3 citations
This work introduces an architectural framework that enables robotic control by leveraging generative AI in a reactive, constrained loop enables robust fulfillment of complex intentions in dynamic and unknown environments.
Justus Flerlage, Thorsten Wittkopp, Alexander Acker et al.· Leveraging Applications of F...· 0 citations
Industrial robot task planning in manufacturing environments still relies heavily on manual programming and expert experience, which makes it difficult to respond quickly to flexible production requirements. Large language models (LLMs) provide a promising approach for translating natural language instructions into exe...
This work introduces a unified formalism for proactive robot assistance, organize it into three levels, and provides a framework to address the highest level of unprompted proactive assistance, and presents a method, GAP, that instantiates the framework, learning from passive observation to anticipate user goals and ac...
Building reliable robot capabilities across diverse tasks requires substantial human effort to develop and maintain skills, design rewards, and integrate perception with control. We present Reconstruct, Practice, Go Real (RPG), a framework for autonomous improvement of robot execution systems without updating model wei...
Yen-Jen Wang, Hao-Zhe Jiang, Shu-Ying Deng et al.· 12 citations
This work formalizes the reasoning-execution boundary as a typed contract and constrains language-level decisions through schema-validated tool calls defined by the Model Context Protocol, rejecting malformed commands before they reach the robot.
Wen-Hao Hong, Lan Wei, Dan-Dan Zhang· 1 citation
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 18, 2026
Founded by Anthony Jules ’92, SM ’92 and Professor Emeritus Rodney Brooks, Robust AI’s robots are designed to collaborate with humans in factories and warehouses.
MIT News · Artificial Intelligence· news.mit.eduNov 20, 2026
Senior Research Scientist Jim Glass discusses how his MIT-IBM Computing Research Lab collaboration connects perception and language, and addresses the shortcoming of current models.