Skip to content

Evaluating LLMs in ROS robotic software code generation

Aug 2026 · Empirical Software Engineering · Vol 32 · 0 citations · 44 references
Computer Science

TL;DR

This work constructs ROSDevEval, a specialized benchmark comprising 240 real-world ROS programming tasks, and conducts a comprehensive empirical study to evaluate four state-of-the-art LLMs alongside a specialized code assistant (GitHub Copilot), revealing a severe domain capability gap.

View source

Similar papers

How does ROS-based Robot Software Evolve? An Empirical Study on Architectural Evolution

A differential, cross-language architectural recovery tool is developed that extracts components, connections, and configurations from source code and applies it to 294 repositories, with 1728 packages across 9914 releases, spanning ROS 1 and ROS 2 in C/C++, Python, and XML.

Paulo Canelas, Bradley Schmerl, Alcides Fonseca et al. · 0 citations
#machine learning Conference Jul 2026

Automated PLC Programming Using a Multi-Task LLM System

Programmable Logic Controllers (PLCs) are central to industrial automation, yet their programming remains largely manual and time-consuming. While Large Language Models (LLMs) excel at general-purpose code generation, their use in industrial control is limited by tight coupling with physical hardware, constrained I/O s...

Teresa Cacciapaglia, D. Ruffino, D. Naso et al. · 0 citations
#artificial intelligence Preprint Aug 2026

MaCoPlanner: LLM-Assisted Manual-Compiled Task Planning with Proactive Safety Verification for Robotic Industrial Panel Operation

MaCoPlanner is presented, a task-planning framework built on knowledge compiled from equipment manuals that converts equipment manuals into a typed intermediate representation, retrieves task- and state-relevant evidence, and uses it to support plan generation.

Gui-Peng Xin, Jiahe Xua, Mohammad Deghat et al. · 0 citations
#small language model Preprint Aug 2026

ROS2SmolVLA: Enabling Small Vision-Language-Action Models for Integration into Industrial-Grade Lightweight Robots

This work adapts Hugging Face's SmolVLA for Universal Robots lightweight robots, and releases the open-source repository ROS2SmolVLA that implements an interface for ROS 2 to SmolVLA, and makes it applicable for industrial-grade hardware.

Nils Mandischer, Noah Böckmann, Ludwig Holl et al. · 0 citations
Jul 2026

Revisiting TuRTLe: A Comprehensive Evaluation of LLMs for RTL Generation

TuRTLe, a unified evaluation framework designed to systematically assess LLMs across key RTL generation tasks, is proposed, enabling a comprehensive assessment of LLM performance in syntax correctness, functional correctness, synthesis, PPA optimization, and exact line completion.

Miquel Albertí-Binimelis, Cristian Gutierrez-Gomez, D. García-Gasulla et al. · 0 citations
Jul 2026

Contract-Grounded Behavior Tree Synthesis via Coding Agents

A contract-grounded BT synthesis architecture in which a coding agent queries a robot-side Model Context Protocol (MCP) server to retrieve an explicit contract consisting of a skill library, permitted BT operators, and optional BT composition templates, before synthesizing a BT for validation and execution is proposed.

Jonathan Salfity, R. Anderson, Mitchell W. Pryor · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.