REFLECT: Summarizing Robot Experiences for Failure Explanation and
Correction
- URL: http://arxiv.org/abs/2306.15724v4
- Date: Mon, 16 Oct 2023 21:04:57 GMT
- Title: REFLECT: Summarizing Robot Experiences for Failure Explanation and
Correction
- Authors: Zeyi Liu, Arpit Bahety, Shuran Song
- Abstract summary: REFLECT is a framework which queries Large Language Models for failure reasoning based on a hierarchical summary of robot past experiences.
We show that REFLECT is able to generate informative failure explanations that assist successful correction planning.
- Score: 28.015693808520496
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The ability to detect and analyze failed executions automatically is crucial
for an explainable and robust robotic system. Recently, Large Language Models
(LLMs) have demonstrated strong reasoning abilities on textual inputs. To
leverage the power of LLMs for robot failure explanation, we introduce REFLECT,
a framework which queries LLM for failure reasoning based on a hierarchical
summary of robot past experiences generated from multisensory observations. The
failure explanation can further guide a language-based planner to correct the
failure and complete the task. To systematically evaluate the framework, we
create the RoboFail dataset with a variety of tasks and failure scenarios. We
demonstrate that the LLM-based framework is able to generate informative
failure explanations that assist successful correction planning.
Related papers
- LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
Large Language Models (LLMs) equipped with extensive world knowledge and strong reasoning skills can tackle diverse tasks across domains.
We propose LLaRA: Large Language and Robotics Assistant, a framework which formulates robot action policy as conversations.
arXiv Detail & Related papers (2024-06-28T17:59:12Z) - Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning [53.6472920229013]
Large Language Models (LLMs) have demonstrated impressive capability in many natural language tasks.
LLMs are prone to produce errors, hallucinations and inconsistent statements when performing multi-step reasoning.
We introduce Q*, a framework for guiding LLMs decoding process with deliberative planning.
arXiv Detail & Related papers (2024-06-20T13:08:09Z) - Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Planners [10.746821861109176]
Large Language Models (LLMs) have witnessed remarkable performance as zero-shot task planners for robotic tasks.
However, the open-loop nature of previous works makes LLM-based planning error-prone and fragile.
In this work, we introduce a framework for closed-loop LLM-based planning called KnowLoop, backed by an uncertainty-based MLLMs failure detector.
arXiv Detail & Related papers (2024-06-01T12:52:06Z) - Self-Corrected Multimodal Large Language Model for End-to-End Robot Manipulation [30.54275273155153]
Multimodal Large Language Models (MLLMs) have shown promise in visual instruction following.
We introduce a Self-Corrected (SC)-MLLM, equipping our model not only to predict end-effector poses but also to autonomously recognize and correct failure actions.
SC-MLLM agent significantly improve manipulation accuracy compared to previous state-of-the-art robotic MLLM (ManipLLM)
arXiv Detail & Related papers (2024-05-27T17:58:48Z) - MOKA: Open-Vocabulary Robotic Manipulation through Mark-Based Visual
Prompting [106.53784213239479]
We present MOKA (Marking Open-vocabulary Keypoint Affordances), an approach that employs vision language models to solve robotic manipulation tasks.
At the heart of our approach is a compact point-based representation of affordance and motion that bridges the VLM's predictions on RGB images and the robot's motions in the physical world.
We evaluate and analyze MOKA's performance on a variety of manipulation tasks specified by free-form language descriptions.
arXiv Detail & Related papers (2024-03-05T18:08:45Z) - Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity [0.659529078336196]
Large language models (LLMs) exhibit advanced reasoning skills, enabling robots to comprehend natural language instructions.
LLMs hallucination may result in robots executing plans that are misaligned with user goals or, in extreme cases, unsafe.
This paper explores the concept of introspective planning as a systematic method for guiding LLMs in forming uncertainty-aware plans for robotic task execution.
arXiv Detail & Related papers (2024-02-09T16:40:59Z) - Interactive Planning Using Large Language Models for Partially
Observable Robotics Tasks [54.60571399091711]
Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks.
We present an interactive planning technique for partially observable tasks using LLMs.
arXiv Detail & Related papers (2023-12-11T22:54:44Z) - A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning.
Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
arXiv Detail & Related papers (2023-11-14T07:13:10Z) - Vision-Language Foundation Models as Effective Robot Imitators [48.73027330407576]
We derive a vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo.
By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control.
arXiv Detail & Related papers (2023-11-02T16:34:33Z) - Interactively Robot Action Planning with Uncertainty Analysis and Active
Questioning by Large Language Model [6.695536752781623]
The Large Language Model (LLM) to robot action planning has been actively studied.
The instructions given to the LLM by natural language may include ambiguity and lack of information depending on the task context.
We propose an interactive robot action planning method that allows the LLM to analyze and gather missing information by asking questions to humans.
arXiv Detail & Related papers (2023-08-30T00:54:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.