VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought

• URL: http://arxiv.org/abs/2406.14596v4
• Date: Fri, 22 Nov 2024 07:43:21 GMT
• Title: VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought
• Authors: Gabriel Sarch, Lawrence Jang, Michael J. Tarr, William W. Cohen, Kenneth Marino, Katerina Fragkiadaki,
• Abstract summary: Large-scale generative language and vision-language models (LLMs and VLMs) excel in few-shot in-context learning for decision making and instruction following. We propose In-Context Abstraction Learning (ICAL), a method that builds a memory of multimodal experience from sub-optimal demonstrations and human feedback.
• Score: 38.03704123835915
• License:
• Abstract: Large-scale generative language and vision-language models (LLMs and VLMs) excel in few-shot in-context learning for decision making and instruction following. However, they require high-quality exemplar demonstrations in their context window. In this work, we ask: Can LLMs and VLMs generate their own examples from generic, sub-optimal demonstrations? We propose In-Context Abstraction Learning (ICAL), a method that builds a memory of multimodal experience from sub-optimal demonstrations and human feedback. Given a task demonstration that may contain inefficiencies or mistakes, a VLM abstracts the trajectory into a generalized program of thoughts by correcting inefficient actions and annotating cognitive abstractions: causal relationships, object state changes, temporal subgoals, and task-relevant visual elements. These programs of thought are iteratively improved through human feedback while the agent executes the trajectory in a similar environment. The resulting examples significantly improve decision-making in retrieval-augmented LLM and VLM agents. Moreover, as the agent's library of examples grows, it becomes more efficient, relying less on human feedback and requiring fewer environment interactions per demonstration. Our ICAL agent surpasses the SOTA in dialogue-based instruction following in TEACh, multimodal web agents in VisualWebArena, and action anticipation in Ego4D. In TEACh, we achieve a 12.6% improvement in goal-condition success. In VisualWebArena, our task success rate improves over few-shot GPT4V. In Ego4D action forecasting, we improve over few-shot GPT-4V and remain competitive with supervised models. We show finetuning our retrieval-augmented in-context agent yields additional improvements. Our approach significantly reduces reliance on manual prompt engineering and consistently outperforms in-context learning from action plans that lack such programs of thought.

Related papers

• Sparse Rewards Can Self-Train Dialogue Agents [22.799506097310008]
  We introduce a novel self-improvement paradigm that empowers LLM agents to autonomously enhance their performance without external human feedback. We present ToolWOZ, a sparse reward tool-calling simulation environment derived from MultiWOZ. We demonstrate that models trained with JOSH, both small and frontier, significantly improve tool-based interactions while preserving general model capabilities across diverse benchmarks.
  arXiv  Detail & Related papers  (2024-09-06T21:00:57Z)
• Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning [6.06616040517684]
  DAAG hindsight relabels the agent's past experience by using diffusion models to transform videos. Large language model orchestrates this autonomous process without requiring human supervision. Results show that DAAG improves learning of reward detectors, transferring past experience, and acquiring new tasks.
  arXiv  Detail & Related papers  (2024-07-30T13:01:31Z)
• Large Vision-Language Models as Emotion Recognizers in Context Awareness [14.85890824622433]
  Context-aware emotion recognition (CAER) is a complex and significant task that requires perceiving emotions from various contextual cues. Previous approaches primarily focus on designing sophisticated architectures to extract emotional cues from images. This paper systematically explore the potential of leveraging Large Vision-Language Models (LVLMs) to empower the CAER task.
  arXiv  Detail & Related papers  (2024-07-16T01:28:06Z)
• Show, Don't Tell: Aligning Language Models with Demonstrated Feedback [54.10302745921713]
  Demonstration ITerated Task Optimization (DITTO) directly aligns language model outputs to a user's demonstrated behaviors. We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts.
  arXiv  Detail & Related papers  (2024-06-02T23:13:56Z)
• Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning [79.38140606606126]
  We propose an algorithmic framework that fine-tunes vision-language models (VLMs) with reinforcement learning (RL) Our framework provides a task description and then prompts the VLM to generate chain-of-thought (CoT) reasoning. We demonstrate that our proposed framework enhances the decision-making capabilities of VLM agents across various tasks.
  arXiv  Detail & Related papers  (2024-05-16T17:50:19Z)
• Exploring the Transferability of Visual Prompting for Multimodal Large Language Models [47.162575147632396]
  Transferable Visual Prompting (TVP) is a simple and effective approach to generate visual prompts that can transfer to different models and improve their performance on downstream tasks after trained on only one model. We introduce two strategies to address the issue of cross-model feature corruption of existing visual prompting methods and enhance the transferability of the learned prompts.
  arXiv  Detail & Related papers  (2024-04-17T09:39:07Z)
• Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement [93.73648674743097]
  Visual program synthesis is a promising approach to exploit the reasoning abilities of large language models for compositional computer vision tasks. Previous work has used few-shot prompting with frozen LLMs to synthesize visual programs. No dataset of visual programs for training exists, and acquisition of a visual program dataset cannot be easily crowdsourced.
  arXiv  Detail & Related papers  (2024-04-06T13:25:00Z)
• VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understanding [65.12464615430036]
  This paper introduces a Video Understanding and Reasoning Framework (VURF) based on the reasoning power of Large Language Models (LLMs) Ours is a novel approach to extend the utility of LLMs in the context of video tasks. We harness their contextual learning capabilities to generate executable visual programs for video understanding.
  arXiv  Detail & Related papers  (2024-03-21T18:00:00Z)
• Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning [49.92517970237088]
  We tackle the problem of training a robot to understand multimodal prompts. This type of task poses a major challenge to robots' capability to understand the interconnection and complementarity between vision and language signals. We introduce an effective framework that learns a policy to perform robot manipulation with multimodal prompts.
  arXiv  Detail & Related papers  (2023-10-14T22:24:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.