LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence

• URL: http://arxiv.org/abs/2405.17424v1
• Date: Mon, 27 May 2024 17:59:32 GMT
• Title: LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
• Authors: Zhuoling Li, Xiaogang Xu, Zhenhua Xu, SerNam Lim, Hengshuang Zhao,
• Abstract summary: We introduce the large auto-regressive model (LARM) for embodied agents. LARM uses both text and multi-view images as input and predicts subsequent actions in an auto-regressive manner. Adopting a two-phase training regimen, LARM successfully harvests enchanted equipment in Minecraft.
• Score: 68.27280750612204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Due to the need to interact with the real world, embodied agents are required to possess comprehensive prior knowledge, long-horizon planning capability, and a swift response speed. Despite recent large language model (LLM) based agents achieving promising performance, they still exhibit several limitations. For instance, the output of LLMs is a descriptive sentence, which is ambiguous when determining specific actions. To address these limitations, we introduce the large auto-regressive model (LARM). LARM leverages both text and multi-view images as input and predicts subsequent actions in an auto-regressive manner. To train LARM, we develop a novel data format named auto-regressive node transmission structure and assemble a corresponding dataset. Adopting a two-phase training regimen, LARM successfully harvests enchanted equipment in Minecraft, which demands significantly more complex decision-making chains than the highest achievements of prior best methods. Besides, the speed of LARM is 6.8x faster.

Related papers

• DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution [114.61347672265076]
  Development of MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. We propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR) that automatically adjusts the size of the activated MLLM. DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  arXiv  Detail & Related papers  (2024-11-04T18:26:08Z)
• Language Models can Self-Lengthen to Generate Long Texts [74.96074422345806]
  This paper introduces an innovative iterative training framework called Self-Lengthen. It leverages only the intrinsic knowledge and skills of Large Language Models without the need for auxiliary data or proprietary models. Experiments on benchmarks and human evaluations show that Self-Lengthen outperforms existing methods in long-text generation.
  arXiv  Detail & Related papers  (2024-10-31T13:47:10Z)
• Two are better than one: Context window extension with multi-grained self-injection [111.1376461868317]
  SharedLLM is a novel approach grounded in the design philosophy of multi-grained context compression and query-aware information retrieval. We introduce a specialized tree-style data structure to efficiently encode, store and retrieve multi-grained contextual information for text chunks.
  arXiv  Detail & Related papers  (2024-10-25T06:08:59Z)
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline. Recent works have started exploiting large language models (LLM) to lessen such burden. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens [15.566726645722657]
  We propose a novel framework specifically designed for speculative sampling. Within this framework, we introduce a lightweight draft model that effectively utilizes previously generated tokens to predict subsequent words. We demonstrate impressive results, achieving an average latency speedup ratio of 2.7x compared to the vanilla auto-regressive decoding approach.
  arXiv  Detail & Related papers  (2024-02-24T08:10:39Z)
• Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding [11.832919020149891]
  This research aims to accelerate the inference speed of large language models (LLMs) with billions of parameters. We propose textbfSmart textbfParallel textbfAuto-textbfCorrect dtextbfEcoding (SPACE)
  arXiv  Detail & Related papers  (2024-02-19T03:39:10Z)
• Speculative Decoding with Big Little Decoder [108.95187338417541]
  Big Little Decoder (BiLD) is a framework that can improve inference efficiency and latency for a wide range of text generation applications. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture.
  arXiv  Detail & Related papers  (2023-02-15T18:55:29Z)
• Capture Salient Historical Information: A Fast and Accurate Non-Autoregressive Model for Multi-turn Spoken Language Understanding [18.988599232838766]
  Existing work increases inference speed by designing non-autoregressive models for single-turn Spoken Language Understanding tasks. We propose a novel model for multi-turn SLU named Salient History Attention with Layer-Refined Transformer (SHA-LRT) SHA captures historical information for the current dialogue from both historical utterances and results via a well-designed history-attention mechanism.
  arXiv  Detail & Related papers  (2022-06-24T10:45:32Z)
• An Effective Non-Autoregressive Model for Spoken Language Understanding [15.99246711701726]
  We propose a novel non-autoregressive Spoken Language Understanding model named Layered-Refine Transformer. With SLG, the non-autoregressive model can efficiently obtain dependency information during training and spend no extra time in inference. Experiments on two public datasets indicate that our model significantly improves SLU performance (1.5% on Overall accuracy) while substantially speed up (more than 10 times) the inference process.
  arXiv  Detail & Related papers  (2021-08-16T10:26:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.