Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation

• URL: http://arxiv.org/abs/2604.18223v1
• Date: Mon, 20 Apr 2026 13:09:13 GMT
• Title: Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation
• Abstract summary: We introduce State-Entangled Environment-Guided Instruction Understanding (S-EGIU)<n>S-EGIU is a decision-relevant, token-level instruction state that evolves step by step conditioned on the agent's perceptual state.<n>It delivers strong performance on several key metrics, including a +2.68% SPL gain on REVERIE Test Unseen.
• Score: 38.77209165510599
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-and-Language Navigation requires agents to follow natural-language instructions in visually changing environments. A central challenge is the dynamic entanglement between language and observations: the meaning of instruction shifts as the agent's field of view and spatial context evolve. However, many existing models encode the instruction as a static global representation, limiting their ability to adapt instruction meaning to the current visual context. We therefore model instruction understanding as an Instruction-as-State variable: a decision-relevant, token-level instruction state that evolves step by step conditioned on the agent's perceptual state, where the perceptual state denotes the observation-grounded navigation context at each step. To realize this principle, we introduce State-Entangled Environment-Guided Instruction Understanding (S-EGIU), a coarse-to-fine framework for state-conditioned segment activation and token-level semantic refinement. At the coarse level, S-EGIU activates the instruction segment whose semantics align with the current observation. At the fine level, it refines the activated segment through observation-guided token grounding and contextual modeling, sharpening its internal semantics under the current observation. Together, these stages maintain an instruction state that is continuously updated according to the agent's perceptual state during navigation. S-EGIU delivers strong performance on several key metrics, including a +2.68% SPL gain on REVERIE Test Unseen, and demonstrates consistent efficiency gains across multiple VLN benchmarks, underscoring the value of dynamic instruction--perception entanglement.

Related papers

• Select-to-Act: Hierarchical Reinforcement Learning via Adaptive Language Guidance [21.023547622557775]
  Reinforcement Learning (RL) has been widely applied to sequential decisionmaking, yet it often suffers from poor sample efficiency due to costly interactions with the environment.<n>We propose emphHierarchical Reinforcement Learning with Language Instructions (HRLLI), a hierarchical RL framework that explicitly models natural-language instructions as dynamically selectable semantic guidance during decisionmaking.
  arXiv  Detail & Related papers  (2026-06-21T06:15:42Z)
• AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation [78.80536515102305]
  Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment.<n>We propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism.<n>Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training.
  arXiv  Detail & Related papers  (2026-05-21T17:58:26Z)
• Language-Grounded Decoupled Action Representation for Robotic Manipulation [78.42228162226839]
  We propose the Language-Grounded Decoupled Action Representation (LaDA) framework to connect perception and control.<n>LaDA introduces a fine-grained intermediate layer of three interpretable action primitives--translation, rotation, and gripper control--providing explicit semantic structure for low-level actions.<n>It further employs a semantic-guided soft-label contrastive learning objective to align similar action primitives across tasks, enhancing generalization and motion consistency.
  arXiv  Detail & Related papers  (2026-03-13T13:08:26Z)
• \textsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents [43.5771856761934]
  Vision-and-Language Navigation (VLN) poses significant challenges for agents to interpret natural language instructions and navigate complex 3D environments.<n>We propose SkillNav, a modular framework that introduces structured, skill-based reasoning into Transformer-based VLN agents.
  arXiv  Detail & Related papers  (2025-08-11T05:50:30Z)
• SituationalLLM: Proactive language models with scene awareness for dynamic, contextual task guidance [13.155859243167619]
  We present SituationalLLM, a novel approach that integrates structured scene information into an large language model.<n>By encoding objects, attributes, and relationships in a custom Scene Graph Language, SituationalLLM actively identifies gaps in environmental context and seeks clarifications during user interactions.<n> Experimental results indicate that SituationalLLM outperforms generic LLM baselines in task specificity, reliability, and adaptability.
  arXiv  Detail & Related papers  (2024-06-19T07:42:48Z)
• Actional Atomic-Concept Learning for Demystifying Vision-Language Navigation [124.07372905781696]
  Actional Atomic-Concept Learning (AACL) maps visual observations to actional atomic concepts for facilitating the alignment. AACL establishes new state-of-the-art results on both fine-grained (R2R) and high-level (REVERIE and R2R-Last) VLN benchmarks.
  arXiv  Detail & Related papers  (2023-02-13T03:08:05Z)
• Graph based Environment Representation for Vision-and-Language Navigation in Continuous Environments [20.114506226598508]
  Vision-and-Language Navigation in Continuous Environments (VLN-CE) is a navigation task that requires an agent to follow a language instruction in a realistic environment. We propose a new environment representation in order to solve the above problems.
  arXiv  Detail & Related papers  (2023-01-11T08:04:18Z)
• Contrastive Instruction-Trajectory Learning for Vision-Language Navigation [66.16980504844233]
  A vision-language navigation (VLN) task requires an agent to reach a target with the guidance of natural language instruction. Previous works fail to discriminate the similarities and discrepancies across instruction-trajectory pairs and ignore the temporal continuity of sub-instructions. We propose a Contrastive Instruction-Trajectory Learning framework that explores invariance across similar data samples and variance across different ones to learn distinctive representations for robust navigation.
  arXiv  Detail & Related papers  (2021-12-08T06:32:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.