Creating and Repairing Robot Programs in Open-World Domains

• URL: http://arxiv.org/abs/2410.18893v1
• Date: Thu, 24 Oct 2024 16:30:14 GMT
• Title: Creating and Repairing Robot Programs in Open-World Domains
• Abstract summary: We propose a system which traces the execution of a program up until error, and then runs an LLM-produced recovery program that minimizes repeated actions. We create a benchmark consisting of eleven tasks with various error conditions that require the generation of a recovery program. We compare the efficiency of the recovery program to a plan built with an oracle that has foreknowledge of future errors.
• Score: 8.93008148936798
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Using Large Language Models (LLMs) to produce robot programs from natural language has allowed for robot systems that can complete a higher diversity of tasks. However, LLM-generated programs may be faulty, either due to ambiguity in instructions, misinterpretation of the desired task, or missing information about the world state. As these programs run, the state of the world changes and they gather new information. When a failure occurs, it is important that they recover from the current world state and avoid repeating steps that they they previously completed successfully. We propose RoboRepair, a system which traces the execution of a program up until error, and then runs an LLM-produced recovery program that minimizes repeated actions. To evaluate the efficacy of our system, we create a benchmark consisting of eleven tasks with various error conditions that require the generation of a recovery program. We compare the efficiency of the recovery program to a plan built with an oracle that has foreknowledge of future errors.

Related papers

• LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models [48.57715789642472]
  We introduce LIBERO-Recover Benchmark, a large scale benchmark for failure recovery in robotic manipulation.<n>We collect real execution failures from SOTA embodied models and construct 1,000+ scenarios across four recovery levels.<n>We evaluate four core capabilities: spatial understanding, object structure reasoning, interaction understanding, and topological reasoning.
  arXiv  Detail & Related papers  (2026-09-04T14:15:27Z)
• SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming [15.199350753007339]
  SkillComposer is an interactive natural-language robot programming system for simulation environments.<n>Successful programs are stored and processed by an online library-learning algorithm.<n>Results show that evaluator-guided generation and learned abstractions improve success rates and usability.
  arXiv  Detail & Related papers  (2026-08-14T23:49:54Z)
• EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration [75.13534797264485]
  We show that egocentric human data capturing failure recovery processes provides a scalable alternative to robot teleoperation.<n>By efficiently arranging task-level failure configurations and recording short recovery segments, human operators can generate more than 10x as much valid recovery data per hour.
  arXiv  Detail & Related papers  (2026-07-22T04:44:26Z)
• Reducing Mental Workload through On-Demand Human Assistance for Physical Action Failures in LLM-based Multi-Robot Coordination [6.763690463901024]
  In this study, we propose a human-in-the-loop framework that integrates remote error resolution into multi-robot planning.<n>In this method, robots execute tasks autonomously; however, when an irrecoverable failure occurs, the LLM requests assistance from an operator.<n>For easily collectable items, it achieved task progress equivalent to full remote control.
  arXiv  Detail & Related papers  (2026-03-30T08:21:04Z)
• Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models [53.20969621498248]
  We propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures.<n>We construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail.<n>We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection.
  arXiv  Detail & Related papers  (2025-12-01T17:57:27Z)
• An LLM-powered Natural-to-Robotic Language Translation Framework with Correctness Guarantees [40.31829665216206]
  Large Language Models (LLM) are increasingly being deployed in robotics to generate robot control programs for specific user tasks.<n>This paper introduces a natural-robotic language translation framework that provides correctness verification for generated control programs.
  arXiv  Detail & Related papers  (2025-08-26T14:32:49Z)
• Self-Corrective Task Planning by Inverse Prompting with Large Language Models [9.283971287618261]
  We introduce InversePrompt, a novel self-corrective task planning approach. Our method incorporates reasoning steps to provide clear, interpretable feedback. Results on benchmark datasets show an average 16.3% higher success rate over existing LLM-based task planning methods.
  arXiv  Detail & Related papers  (2025-03-10T13:35:51Z)
• Autellix: An Efficient Serving Engine for LLM Agents as General Programs [59.673243129044465]
  Large language model (LLM) applications are evolving beyond simple chatbots into dynamic, general-purpose agentic programs. Existing LLM serving systems ignore dependencies between programs and calls, missing significant opportunities for optimization. We introduce Autellix, an LLM serving system that treats programs as first-class citizens to minimize their end-to-end latencies.
  arXiv  Detail & Related papers  (2025-02-19T18:59:30Z)
• Interactive and Expressive Code-Augmented Planning with Large Language Models [62.799579304821826]
  Large Language Models (LLMs) demonstrate strong abilities in common-sense reasoning and interactive decision-making. Recent techniques have sought to structure LLM outputs using control flow and other code-adjacent techniques to improve planning performance. We propose REPL-Plan, an LLM planning approach that is fully code-expressive and dynamic.
  arXiv  Detail & Related papers  (2024-11-21T04:23:17Z)
• DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution [114.61347672265076]
  Development of MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. We propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR) that automatically adjusts the size of the activated MLLM. DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  arXiv  Detail & Related papers  (2024-11-04T18:26:08Z)
• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• Joint Verification and Refinement of Language Models for Safety-Constrained Planning [25.390607788572535]
  Large language models can generate programs from natural language descriptions to execute robotic tasks.<n>However, these generated programs often contain errors that violate externally given task specifications.<n>We develop a method that converts generated robot programs into an automaton-based representation and verifies them against task-relevant safety specifications.
  arXiv  Detail & Related papers  (2024-10-18T21:16:30Z)
• COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models [49.24666980374751]
  COHERENT is a novel LLM-based task planning framework for collaboration of heterogeneous multi-robot systems. A Proposal-Execution-Feedback-Adjustment mechanism is designed to decompose and assign actions for individual robots. The experimental results show that our work surpasses the previous methods by a large margin in terms of success rate and execution efficiency.
  arXiv  Detail & Related papers  (2024-09-23T15:53:41Z)
• RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning [19.023560632891467]
  We propose a scalable data generation pipeline that augments expert demonstrations with failure recovery trajectories. We then introduce Rich languAge-guided failure reCovERy (RACER), a supervisor-actor framework. Our experimental results show that RACER outperforms the state-of-the-art Robotic View Transformer on RLbench.
  arXiv  Detail & Related papers  (2024-09-23T02:50:33Z)
• Solving Robotics Problems in Zero-Shot with Vision-Language Models [0.0]
  We introduce Wonderful Team, a multi-agent Vision Large Language Model (VLLM) framework designed to solve robotics problems in a zero-shot regime. In our context, zero-shot means that for a novel environment, we provide a VLLM with an image of the robot's surroundings and a task description. Our system showcases the ability to handle diverse tasks such as manipulation, goal-reaching, and visual reasoning -- all in a zero-shot manner.
  arXiv  Detail & Related papers  (2024-07-26T21:18:57Z)
• Robo-Instruct: Simulator-Augmented Instruction Alignment For Finetuning Code LLMs [42.31298987176411]
  We introduce ROBO-INSTRUCT, which synthesizes task-specific simulation environments on the fly during program execution. ROBO-INSTRUCT integrates an LLM-aided post-processing procedure to refine instructions for better alignment with robot programs.
  arXiv  Detail & Related papers  (2024-05-30T15:47:54Z)
• Peer-aided Repairer: Empowering Large Language Models to Repair Advanced Student Assignments [26.236420215606238]
  We develop a framework called PaR that is powered by the Large Language Model. PaR works in three phases: Peer Solution Selection, Multi-Source Prompt Generation, and Program Repair. The evaluation on Defects4DS and another well-investigated ITSP dataset reveals that PaR achieves a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-04-02T09:12:21Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment [10.322491116030825]
  DoReMi enables Detection and Recovery from Misalignments between plan and execution. Our pipeline can monitor the low-level execution and enable timely recovery if certain plan-execution misalignment occurs.
  arXiv  Detail & Related papers  (2023-07-01T12:51:02Z)
• REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction [28.015693808520496]
  REFLECT is a framework which queries Large Language Models for failure reasoning based on a hierarchical summary of robot past experiences. We show that REFLECT is able to generate informative failure explanations that assist successful correction planning.
  arXiv  Detail & Related papers  (2023-06-27T18:03:15Z)
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models [68.57918965060787]
  Large language models (LLMs) can be used to score potential next actions during task planning. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
  arXiv  Detail & Related papers  (2022-09-22T20:29:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.