ReLEP: A Novel Framework for Real-world Long-horizon Embodied Planning

• URL: http://arxiv.org/abs/2409.15658v1
• Date: Tue, 24 Sep 2024 01:47:23 GMT
• Title: ReLEP: A Novel Framework for Real-world Long-horizon Embodied Planning
• Authors: Siyuan Liu, Jiawei Du, Sicheng Xiang, Zibo Wang, Dingsheng Luo,
• Abstract summary: We present ReLEP, a framework for Real world Long-horizon Embodied Planning. At its core lies a fine-tuned large vision language model that formulates plans as sequences of skill functions. ReLEP can accomplish a wide range of daily tasks and outperforms other state-of-the-art baseline methods.
• Score: 7.668848364013772
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-world long-horizon embodied planning underpins embodied AI. To accomplish long-horizon tasks, agents need to decompose abstract instructions into detailed steps. Prior works mostly rely on GPT-4V for task decomposition into predefined actions, which limits task diversity due to GPT-4V's finite understanding of larger skillsets. Therefore, we present ReLEP, a groundbreaking framework for Real world Long-horizon Embodied Planning, which can accomplish a wide range of daily tasks. At its core lies a fine-tuned large vision language model that formulates plans as sequences of skill functions according to input instruction and scene image. These functions are selected from a carefully designed skill library. ReLEP is also equipped with a Memory module for plan and status recall, and a Robot Configuration module for versatility across robot types. In addition, we propose a semi-automatic data generation pipeline to tackle dataset scarcity. Real-world off-line experiments across eight daily embodied tasks demonstrate that ReLEP is able to accomplish long-horizon embodied tasks and outperforms other state-of-the-art baseline methods.

Related papers

• Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks [50.27313829438866]
  Plan-Seq-Learn (PSL) is a modular approach that uses motion planning to bridge the gap between abstract language and learned low-level control. PSL achieves success rates of over 85%, out-performing language-based, classical, and end-to-end approaches.
  arXiv  Detail & Related papers  (2024-05-02T17:59:31Z)
• RoboGPT: an intelligent agent of making embodied long-term decisions for daily instruction tasks [13.29302304547683]
  Large Language Models (LLMs) in natural language processing have inspired efforts to use LLMs in complex robot planning. We propose a RoboGPT agent for making embodied long-term decisions for daily tasks. The proposed RoboGPT agent outperforms SOTA methods on the ALFRED daily tasks.
  arXiv  Detail & Related papers  (2023-11-27T09:20:23Z)
• Generalizable Long-Horizon Manipulations with Large Language Models [91.740084601715]
  This work introduces a framework harnessing the capabilities of Large Language Models (LLMs) to generate primitive task conditions for generalizable long-horizon manipulations. We create a challenging robotic manipulation task suite based on Pybullet for long-horizon task evaluation.
  arXiv  Detail & Related papers  (2023-10-03T17:59:46Z)
• Embodied Task Planning with Large Language Models [86.63533340293361]
  We propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint. During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations. Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin.
  arXiv  Detail & Related papers  (2023-07-04T17:58:25Z)
• Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planning [45.51792981370957]
  We investigate the applicability of a smaller class of large language models (LLMs) in robotic task planning by learning to decompose tasks into subgoal specifications for a planner to execute sequentially. Our method grounds the input of the LLM on the domain that is represented as a scene graph, enabling it to translate human requests into executable robot plans. Our findings suggest that the knowledge stored in an LLM can be effectively grounded to perform long-horizon task planning, demonstrating the promising potential for the future application of neuro-symbolic planning methods in robotics.
  arXiv  Detail & Related papers  (2023-05-12T18:14:32Z)
• Efficient Learning of High Level Plans from Play [57.29562823883257]
  We present Efficient Learning of High-Level Plans from Play (ELF-P), a framework for robotic learning that bridges motion planning and deep RL. We demonstrate that ELF-P has significantly better sample efficiency than relevant baselines over multiple realistic manipulation tasks.
  arXiv  Detail & Related papers  (2023-03-16T20:09:47Z)
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models [68.57918965060787]
  Large language models (LLMs) can be used to score potential next actions during task planning. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
  arXiv  Detail & Related papers  (2022-09-22T20:29:49Z)
• Long-Horizon Planning and Execution with Functional Object-Oriented Networks [79.94575713911189]
  We introduce the idea of exploiting object-level knowledge as a FOON for task planning and execution. Our approach automatically transforms FOON into PDDL and leverages off-the-shelf planners, action contexts, and robot skills. We demonstrate our approach on long-horizon tasks in CoppeliaSim and show how learned action contexts can be extended to never-before-seen scenarios.
  arXiv  Detail & Related papers  (2022-07-12T19:29:35Z)
• Hierarchies of Planning and Reinforcement Learning for Robot Navigation [22.08479169489373]
  In many navigation tasks, high-level (HL) task representations, like a rough floor plan, are available. Previous work has demonstrated efficient learning by hierarchal approaches consisting of path planning in the HL representation. This work proposes a novel hierarchical framework that utilizes a trainable planning policy for the HL representation.
  arXiv  Detail & Related papers  (2021-09-23T07:18:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.