APPROVE: Visual End-User-in-the-Loop Robot Programming with LLMs
- URL: http://arxiv.org/abs/2608.19281v1
- Date: Wed, 19 Aug 2026 08:22:13 GMT
- Title: APPROVE: Visual End-User-in-the-Loop Robot Programming with LLMs
- Abstract summary: We present APPROVE (AI-Powered Programming for Robots with Visual End-User Feedback), an LLM-based multi-modal end-user programming framework.<n> Generated programs are visualized using a block-based interface in Blockly, allowing users to confirm, modify, or reject them before execution.<n>Our approach contributes a human-centered design for LLM-based robot programming that emphasizes user trust, intent alignment, and reusability.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Programming robots remains challenging for non-experts, as traditional methods require expert knowledge and even block-based interfaces often lack flexibility. Recent work has explored Large Language Models (LLMs) to automatically generate robot programs from natural language, but these systems remain limited by a lack of transparency, missing mechanisms to ensure alignment with user intent, and little support for reuse. We present APPROVE (AI-Powered Programming for Robots with Visual End-User Feedback), an LLM-based multi-modal end-user programming framework that integrates natural language input with a block-based interface and an explicit user confirmation step. Generated programs are visualized using a block-based interface in Blockly, allowing users to confirm, modify, or reject them before execution. Confirmed functions are stored in a library for reuse, gradually building a set of reliable program components. Our approach contributes a human-centered design for LLM-based robot programming that emphasizes user trust, intent alignment, and reusability.
Related papers
- SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming [15.199350753007339]
SkillComposer is an interactive natural-language robot programming system for simulation environments.<n>Successful programs are stored and processed by an online library-learning algorithm.<n>Results show that evaluator-guided generation and learned abstractions improve success rates and usability.
arXiv Detail & Related papers (2026-08-14T23:49:54Z) - Robo-Blocks: Generative Scaffolding in End-User Design and Programming of Social Robots [32.50504092559475]
Large language models (LLMs) can obscure critical elements of programming and supplant designer intent.<n>We show how Robo-Blocks, a block-based programming environment, offers novice robot programmers generative scaffolding through structured narratives.<n>We present design insights for the effective use of generative scaffolding and its integration into the practice of social-robot programming.
arXiv Detail & Related papers (2026-05-27T08:39:01Z) - RoboCritics: Enabling Reliable End-to-End LLM Robot Programming through Expert-Informed Critics [25.430531368529685]
Large Language Models (LLMs) hold the potential to lower the barrier to robot programming by enabling task specification through natural language.<n>We present RoboCritics, an approach that augments LLM-based robot programming with expert-informed motion-level critics.
arXiv Detail & Related papers (2026-03-06T20:02:53Z) - Generative Interfaces for Language Models [70.25765232527762]
We propose a paradigm in which large language models (LLMs) respond to user queries by proactively generating user interfaces (UIs)<n>Our framework leverages structured interface-specific representations and iterative refinements to translate user queries into task-specific UIs.<n>Results show that generative interfaces consistently outperform conversational ones, with up to a 72% improvement in human preference.
arXiv Detail & Related papers (2025-08-26T17:43:20Z) - Capability-Driven Skill Generation with LLMs: A RAG-Based Approach for Reusing Existing Libraries and Interfaces [40.638726615548954]
We present a method that treats capabilities as contracts for skill implementations and leverages large language models to generate code based on natural language user input.<n>A key feature of our approach is the integration of existing software libraries and interface technologies.<n>We introduce a framework that allows users to incorporate their own libraries and resource interfaces into the code generation process.
arXiv Detail & Related papers (2025-05-06T08:27:04Z) - Skill Discovery for Software Scripting Automation via Offline Simulations with LLMs [63.10710876536337]
We propose an offline simulation framework to curate a software-specific skillset, a collection of verified scripts.<n>Our framework comprises two components: (1) task creation, using top-down functionality and bottom-up API synergy exploration to generate helpful tasks.<n> Experiments with Adobe Illustrator demonstrate that our framework significantly improves automation success rates, reduces response time, and saves runtime token costs.
arXiv Detail & Related papers (2025-04-29T04:03:37Z) - Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming [3.041618201510648]
We introduce Cocobo, a natural language programming system with interactive diagrams powered by large language models (LLMss)
Cocobo employs LLMs to understand users' authoring intentions, generate and explain robot programs, and facilitate the conversion between executable code and flowchart representations.
Our user study shows that Cocobo has a low learning curve, enabling even users with zero coding experience to customize robot programs successfully.
arXiv Detail & Related papers (2024-07-30T10:13:00Z) - ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning [74.58666091522198]
We present a framework for intuitive robot programming by non-experts.
We leverage natural language prompts and contextual information from the Robot Operating System (ROS)
Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface.
arXiv Detail & Related papers (2024-06-28T08:28:38Z) - WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model [92.90127398282209]
This paper investigates the potential of integrating the most recent Large Language Models (LLMs) and existing visual grounding and robotic grasping system.
We introduce the WALL-E (Embodied Robotic WAiter load lifting with Large Language model) as an example of this integration.
We deploy this LLM-empowered system on the physical robot to provide a more user-friendly interface for the instruction-guided grasping task.
arXiv Detail & Related papers (2023-08-30T11:35:21Z) - PADL: Language-Directed Physics-Based Character Control [66.517142635815]
We present PADL, which allows users to issue natural language commands for specifying high-level tasks and low-level skills that a character should perform.
We show that our framework can be applied to effectively direct a simulated humanoid character to perform a diverse array of complex motor skills.
arXiv Detail & Related papers (2023-01-31T18:59:22Z) - GenNI: Human-AI Collaboration for Data-Backed Text Generation [102.08127062293111]
Table2Text systems generate textual output based on structured data utilizing machine learning.
GenNI (Generation Negotiation Interface) is an interactive visual system for high-level human-AI collaboration in producing descriptive text.
arXiv Detail & Related papers (2021-10-19T18:07:07Z) - Learning Adaptive Language Interfaces through Decomposition [89.21937539950966]
We introduce a neural semantic parsing system that learns new high-level abstractions through decomposition.
Users interactively teach the system by breaking down high-level utterances describing novel behavior into low-level steps.
arXiv Detail & Related papers (2020-10-11T08:27:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.