Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft

• URL: http://arxiv.org/abs/2406.17553v1
• Date: Tue, 25 Jun 2024 13:43:24 GMT
• Title: Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft
• Authors: Chalamalasetti Kranti, Sherzod Hakimov, David Schlangen,
• Abstract summary: In the Minecraft Collaborative Building Task, two players collaborate: an Architect (A) provides instructions to a Builder (B) to assemble a specified structure using 3D blocks. We investigate the use of large language models (LLMs) to predict the sequence of actions taken by the Builder.
• Score: 18.256529559741075
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the Minecraft Collaborative Building Task, two players collaborate: an Architect (A) provides instructions to a Builder (B) to assemble a specified structure using 3D blocks. In this work, we investigate the use of large language models (LLMs) to predict the sequence of actions taken by the Builder. Leveraging LLMs' in-context learning abilities, we use few-shot prompting techniques, that significantly improve performance over baseline methods. Additionally, we present a detailed analysis of the gaps in performance for future work

Related papers

• BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues [7.377606500245465]
  The Minecraft Collaborative Building Task (MCBT) provides one such setting to work towards this goal. We focus on the challenging Builder Action Prediction (BAP) subtask of predicting correct action sequences in a multimodal game context. We take a closer look at evaluation and data for the BAP task, discovering key challenges and making significant improvements on both fronts to propose BAP v2, an upgraded version of the task.
  arXiv  Detail & Related papers  (2025-01-18T18:06:03Z)
• APT: Architectural Planning and Text-to-Blueprint Construction Using Large Language Models for Open-World Agents [8.479128275067742]
  We present an advanced Large Language Model (LLM)-driven framework that enables autonomous agents to construct complex structures in Minecraft. By employing chain-of-thought decomposition along with multimodal inputs, the framework generates detailed architectural layouts and blueprints. Our agent incorporates both memory and reflection modules to facilitate lifelong learning, adaptive refinement, and error correction throughout the building process.
  arXiv  Detail & Related papers  (2024-11-26T09:31:28Z)
• TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly [51.29305265324916]
  We propose a class-agnostic tree-transformer framework to predict the sequential assembly actions from input multi-view images. A major challenge of the sequential brick assembly task is that the step-wise action labels are costly and tedious to obtain in practice. We mitigate this problem by leveraging synthetic-to-real transfer learning.
  arXiv  Detail & Related papers  (2024-07-22T14:05:27Z)
• A LLM Benchmark based on the Minecraft Builder Dialog Agent Task [5.555936227537389]
  This work proposes adapting the Minecraft builder task into an LLM benchmark suitable for evaluating LLM ability in spatially orientated tasks. We believe this approach allows us to probe specific strengths and weaknesses of different agents, and test the ability of LLMs in the challenging area of spatial reasoning and vector based math.
  arXiv  Detail & Related papers  (2024-07-17T16:52:23Z)
• Towards Completeness-Oriented Tool Retrieval for Large Language Models [60.733557487886635]
  Real-world systems often incorporate a wide array of tools, making it impractical to input all tools into Large Language Models. Existing tool retrieval methods primarily focus on semantic matching between user queries and tool descriptions. We propose a novel modelagnostic COllaborative Learning-based Tool Retrieval approach, COLT, which captures not only the semantic similarities between user queries and tool descriptions but also takes into account the collaborative information of tools.
  arXiv  Detail & Related papers  (2024-05-25T06:41:23Z)
• Demystifying Instruction Mixing for Fine-tuning Large Language Models [29.69436955342966]
  This study categorizes instructions into three primary types: NLP downstream tasks, coding, and general chat. We find that certain instruction types are more advantageous for specific applications but can negatively impact other areas.
  arXiv  Detail & Related papers  (2023-12-17T18:44:26Z)
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets [75.64181719386497]
  We present CRAFT, a tool creation and retrieval framework for large language models (LLMs) It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning.
  arXiv  Detail & Related papers  (2023-09-29T17:40:26Z)
• CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models [74.22729793816451]
  Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability. We propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems.
  arXiv  Detail & Related papers  (2023-05-23T17:51:52Z)
• Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model [63.66204449776262]
  Instruct2Act is a framework that maps multi-modal instructions to sequential actions for robotic manipulation tasks. Our approach is adjustable and flexible in accommodating various instruction modalities and input types. Our zero-shot method outperformed many state-of-the-art learning-based policies in several tasks.
  arXiv  Detail & Related papers  (2023-05-18T17:59:49Z)
• Learning to Execute Actions or Ask Clarification Questions [9.784428580459776]
  We propose a new builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task.
  arXiv  Detail & Related papers  (2022-04-18T15:36:02Z)
• Multi-Task Learning for Dense Prediction Tasks: A Survey [87.66280582034838]
  Multi-task learning (MTL) techniques have shown promising results w.r.t. performance, computations and/or memory footprint. We provide a well-rounded view on state-of-the-art deep learning approaches for MTL in computer vision.
  arXiv  Detail & Related papers  (2020-04-28T09:15:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.