AIGeN: An Adversarial Approach for Instruction Generation in VLN

• URL: http://arxiv.org/abs/2404.10054v1
• Date: Mon, 15 Apr 2024 18:00:30 GMT
• Title: AIGeN: An Adversarial Approach for Instruction Generation in VLN
• Authors: Niyati Rawal, Roberto Bigazzi, Lorenzo Baraldi, Rita Cucchiara,
• Abstract summary: We propose AIGeN, a novel architecture inspired by Generative Adrial Networks (GANs) that produces meaningful and well-formed synthetic instructions to improve navigation agents' performance. We generate synthetic instructions for 217K trajectories using AIGeN on Habitat-Matterport 3D dataset (HM3D) and show an improvement in the performance of an off-the-shelf VLN method.
• Score: 35.932836008492174
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the last few years, the research interest in Vision-and-Language Navigation (VLN) has grown significantly. VLN is a challenging task that involves an agent following human instructions and navigating in a previously unknown environment to reach a specified goal. Recent work in literature focuses on different ways to augment the available datasets of instructions for improving navigation performance by exploiting synthetic training data. In this work, we propose AIGeN, a novel architecture inspired by Generative Adversarial Networks (GANs) that produces meaningful and well-formed synthetic instructions to improve navigation agents' performance. The model is composed of a Transformer decoder (GPT-2) and a Transformer encoder (BERT). During the training phase, the decoder generates sentences for a sequence of images describing the agent's path to a particular point while the encoder discriminates between real and fake instructions. Experimentally, we evaluate the quality of the generated instructions and perform extensive ablation studies. Additionally, we generate synthetic instructions for 217K trajectories using AIGeN on Habitat-Matterport 3D Dataset (HM3D) and show an improvement in the performance of an off-the-shelf VLN method. The validation analysis of our proposal is conducted on REVERIE and R2R and highlights the promising aspects of our proposal, achieving state-of-the-art performance.

Related papers

• TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings [61.9257731511557]
  We propose Text Guided LLaVA (TG-LLaVA) to optimize vision-language models (VLMs) We use learnable latent embeddings as a bridge to analyze textual instruction and add the analysis results to the vision encoder as guidance. With the guidance of text, the vision encoder can extract text-related features, similar to how humans focus on the most relevant parts of an image when considering a question.
  arXiv  Detail & Related papers  (2024-09-15T00:38:34Z)
• Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning [6.06616040517684]
  DAAG hindsight relabels the agent's past experience by using diffusion models to transform videos. Large language model orchestrates this autonomous process without requiring human supervision. Results show that DAAG improves learning of reward detectors, transferring past experience, and acquiring new tasks.
  arXiv  Detail & Related papers  (2024-07-30T13:01:31Z)
• Vision-and-Language Navigation Generative Pretrained Transformer [0.0]
  Vision-and-Language Navigation Generative Pretrained Transformer (VLN-GPT) Adopts transformer decoder model (GPT2) to model trajectory sequence dependencies, bypassing the need for historical encoding modules. Performance assessments on the VLN dataset reveal that VLN-GPT surpasses complex state-of-the-art encoder-based models.
  arXiv  Detail & Related papers  (2024-05-27T09:42:04Z)
• Learning Vision-and-Language Navigation from YouTube Videos [89.1919348607439]
  Vision-and-language navigation (VLN) requires an embodied agent to navigate in realistic 3D environments using natural language instructions. There are massive house tour videos on YouTube, providing abundant real navigation experiences and layout information. We create a large-scale dataset which comprises reasonable path-instruction pairs from house tour videos and pre-training the agent on it.
  arXiv  Detail & Related papers  (2023-07-22T05:26:50Z)
• Goal-Guided Transformer-Enabled Reinforcement Learning for Efficient Autonomous Navigation [15.501449762687148]
  We present a Goal-guided Transformer-enabled reinforcement learning (GTRL) approach for goal-driven navigation. Our approach motivates the scene representation to concentrate mainly on goal-relevant features, which substantially enhances the data efficiency of the DRL learning process. Both simulation and real-world experimental results manifest the superiority of our approach in terms of data efficiency, performance, robustness, and sim-to-real generalization.
  arXiv  Detail & Related papers  (2023-01-01T07:14:30Z)
• A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning [70.14372215250535]
  Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments. Given the scarcity of human instruction data and limited diversity in the training environments, these agents still struggle with complex language grounding and spatial language understanding. We take 500+ indoor environments captured in densely-sampled 360 degree panoramas, construct navigation trajectories through these panoramas, and generate a visually-grounded instruction for each trajectory. The resulting dataset of 4.2M instruction-trajectory pairs is two orders of magnitude larger than existing human-annotated datasets.
  arXiv  Detail & Related papers  (2022-10-06T17:59:08Z)
• Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigation [172.15808300686584]
  We describe an approach that learns the two tasks simultaneously and exploits their intrinsic correlations to boost the training of each. Our approach improves the performance of various follower models and produces accurate navigation instructions.
  arXiv  Detail & Related papers  (2022-03-30T18:15:26Z)
• Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation [145.84123197129298]
  Language instruction plays an essential role in the natural language grounded navigation tasks. We exploit to train a more robust navigator which is capable of dynamically extracting crucial factors from the long instruction. Specifically, we propose a Dynamic Reinforced Instruction Attacker (DR-Attacker), which learns to mislead the navigator to move to the wrong target.
  arXiv  Detail & Related papers  (2021-07-23T14:11:31Z)
• Generative Adversarial Networks for Annotated Data Augmentation in Data Sparse NLU [0.76146285961466]
  Data sparsity is one of the key challenges associated with model development in Natural Language Understanding. We present our results on boosting NLU model performance through training data augmentation using a sequential generative adversarial network (GAN) Our experiments reveal synthetic data generated using the sequential generative adversarial network provides significant performance boosts across multiple metrics.
  arXiv  Detail & Related papers  (2020-12-09T20:38:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.