LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following

• URL: http://arxiv.org/abs/2310.12344v1
• Date: Wed, 18 Oct 2023 21:43:07 GMT
• Title: LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following
• Abstract summary: We introduce a novel concept of meta-actions to bridge the semantic gap between language instructions and agent's action space. Compared to a strong multi-modal Transformer baseline, we achieve a significant 4.5% absolute gain in success rate in unseen environments.
• Score: 142.25438780561603
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: End-to-end Transformers have demonstrated an impressive success rate for Embodied Instruction Following when the environment has been seen in training. However, they tend to struggle when deployed in an unseen environment. This lack of generalizability is due to the agent's insensitivity to subtle changes in natural language instructions. To mitigate this issue, we propose explicitly aligning the agent's hidden states with the instructions via contrastive learning. Nevertheless, the semantic gap between high-level language instructions and the agent's low-level action space remains an obstacle. Therefore, we further introduce a novel concept of meta-actions to bridge the gap. Meta-actions are ubiquitous action patterns that can be parsed from the original action sequence. These patterns represent higher-level semantics that are intuitively aligned closer to the instructions. When meta-actions are applied as additional training signals, the agent generalizes better to unseen environments. Compared to a strong multi-modal Transformer baseline, we achieve a significant 4.5% absolute gain in success rate in unseen environments of ALFRED Embodied Instruction Following. Additional analysis shows that the contrastive objective and meta-actions are complementary in achieving the best results, and the resulting agent better aligns its states with corresponding instructions, making it more suitable for real-world embodied agents. The code is available at: https://github.com/joeyy5588/LACMA.

Related papers

• Language-Grounded Decoupled Action Representation for Robotic Manipulation [78.42228162226839]
  We propose the Language-Grounded Decoupled Action Representation (LaDA) framework to connect perception and control.<n>LaDA introduces a fine-grained intermediate layer of three interpretable action primitives--translation, rotation, and gripper control--providing explicit semantic structure for low-level actions.<n>It further employs a semantic-guided soft-label contrastive learning objective to align similar action primitives across tasks, enhancing generalization and motion consistency.
  arXiv  Detail & Related papers  (2026-03-13T13:08:26Z)
• Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining [59.2578488860426]
  Heterogeneous multi-modal remote sensing object detection aims to accurately detect objects from diverse sensors.<n>Existing approaches largely adopt a late alignment paradigm, in which modality alignment and task-specific optimization are entangled during downstream fine-tuning.<n>We propose BabelRS, a unified language-pivoted pretraining framework that explicitly decouples modality alignment from downstream task learning.
  arXiv  Detail & Related papers  (2026-03-02T11:38:12Z)
• Language Modeling with Learned Meta-Tokens [15.860245999620409]
  This work introduces a novel approach using meta-tokens, special tokens injected during pre-training, along with a dedicated meta-attention mechanism to guide LMs to use these tokens.<n>We find that data-efficient language model pre-training on fewer than 100B tokens utilizing meta-tokens achieves strong performance on these tasks after fine-tuning.
  arXiv  Detail & Related papers  (2025-09-18T17:38:48Z)
• LinguaFluid: Language Guided Fluid Control via Semantic Rewards in Reinforcement Learning [0.7864304771129751]
  We introduce a semantically aligned reinforcement learning method where rewards are computed by aligning the current state with a target semantic instruction.<n>We show that semantic reward can guide learning to achieve competitive control behavior, even in the absence of hand-crafted reward functions.<n>This framework opens new horizons for aligning agent behavior with natural language goals and lays the groundwork for a more seamless integration of larger language models.
  arXiv  Detail & Related papers  (2025-08-08T03:23:56Z)
• Beyond Syntax: Action Semantics Learning for App Agents [60.56331102288794]
  Action Semantics Learning (ASL) is a learning framework where the learning objective is capturing the semantics of the ground truth actions.<n>ASL significantly improves the accuracy and generalisation of App agents over existing methods.
  arXiv  Detail & Related papers  (2025-06-21T12:08:19Z)
• Abstract Counterfactuals for Language Model Agents [3.7540612510652176]
  Abstract Counterfactuals is a framework that emphasises high-level characteristics of actions and interactions within an environment.<n>We conduct experiments on text-based games and counterfactual text generation, while considering both token-level and latent-space interventions.
  arXiv  Detail & Related papers  (2025-06-03T14:44:26Z)
• MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control [53.20509532671891]
  We introduce MineDreamer, an open-ended embodied agent built upon the Minecraft simulator. We employ a Chain-of-Imagination (CoI) mechanism to envision the step-by-step process of executing instructions. Experiments demonstrate that MineDreamer follows single and multi-step instructions steadily.
  arXiv  Detail & Related papers  (2024-03-18T17:59:42Z)
• Contrastive Instruction Tuning [61.97704869248903]
  We propose Contrastive Instruction Tuning to maximize the similarity between semantically equivalent instruction-instance pairs. Experiments on the PromptBench benchmark show that CoIN consistently improves LLMs' robustness to unseen instructions with variations across character, word, sentence, and semantic levels by an average of +2.5% in accuracy.
  arXiv  Detail & Related papers  (2024-02-17T00:09:32Z)
• Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization [40.45470744120691]
  Self-sUpervised meta-Prompt learning framework with MEta-gradient Regularization for few-shot generalization (SUPMER) This paper proposes a novel Self-sUpervised meta-Prompt learning framework with MEta-gradient Regularization for few-shot generalization (SUPMER)
  arXiv  Detail & Related papers  (2023-03-22T05:04:21Z)
• Entity Divider with Language Grounding in Multi-Agent Reinforcement Learning [28.619845209653274]
  We investigate the use of natural language to drive the generalization of policies in multi-agent settings. We propose a novel framework for language grounding in multi-agent reinforcement learning, entity divider (EnDi) EnDi enables agents to independently learn subgoal division at the entity level and act in the environment based on the associated entities.
  arXiv  Detail & Related papers  (2022-10-25T11:53:52Z)
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents [111.33545170562337]
  We investigate the possibility of grounding high-level tasks, expressed in natural language, to a chosen set of actionable steps. We find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into low-level plans. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions.
  arXiv  Detail & Related papers  (2022-01-18T18:59:45Z)
• Episodic Transformer for Vision-and-Language Navigation [142.6236659368177]
  This paper focuses on addressing two challenges: handling long sequence of subtasks, and understanding complex human instructions. We propose Episodic Transformer (E.T.), a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions. Our approach sets a new state of the art on the challenging ALFRED benchmark, achieving 38.4% and 8.5% task success rates on seen and unseen test splits.
  arXiv  Detail & Related papers  (2021-05-13T17:51:46Z)
• ELLA: Exploration through Learned Language Abstraction [6.809870486883877]
  ELLA is a reward shaping approach that correlates high-level instructions with simpler low-level instructions to enrich the sparse rewards afforded by the environment. ELLA shows a significant gain in sample efficiency across several environments compared to competitive language-based reward shaping and no-shaping methods.
  arXiv  Detail & Related papers  (2021-03-10T02:18:46Z)
• Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation [71.70562795158625]
  Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU) Recent advent of end-to-end neural models, self-supervised via language modeling (LM), and their success on a wide range of LU tasks, questions this belief. We empirically investigate the usefulness of supervised parsing for semantic LU in the context of LM-pretrained transformer networks.
  arXiv  Detail & Related papers  (2020-08-15T21:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.