ViSMoE: Visual-Aware Sparse Mixture-of-Experts for Embodied Referring Expression Grounding

• URL: http://arxiv.org/abs/2608.21878v1
• Date: Sat, 22 Aug 2026 09:43:49 GMT
• Title: ViSMoE: Visual-Aware Sparse Mixture-of-Experts for Embodied Referring Expression Grounding
• Abstract summary: Embodied Referring Expression Grounding is the task of enabling an agent to navigate in real environments.<n>We propose ViSMoE, which equips sparse Mixture-of-Experts with a visual-aware routing policy for the embodied agent.
• Score: 24.39309961528087
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied Referring Expression Grounding is the task of enabling an agent to navigate in real environments and to localize a remote object based on natural language instructions. In this scenario, the agent needs to select one view for navigation at each step and identify a specific object among all candidate objects at the destination. However, most of the previous approaches fail to distinguish between views and objects, instead processing them using the vanilla vision encoder, which results in ambiguous representations of both views and objects. To address the above issues, we propose ViSMoE, which equips sparse Mixture-of-Experts with a visual-aware routing policy for the embodied agent. This framework processes different types of visual information specifically, resulting in discriminative visual representations for both views and objects. Experimental results on REVERIE and SOON datasets demonstrate that ViSMoE outperforms the previous state-of-the-art methods, showing the superiority of our proposed method.

Related papers

• Steerable Visual Representations [72.39044430620977]
  We introduce Steerable Visual Representations, whose global and local features can be steered with natural language.<n>Our method also matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination.
  arXiv  Detail & Related papers  (2026-04-02T17:59:49Z)
• VCA: Vision-Click-Action Framework for Precise Manipulation of Segmented Objects in Target Ambiguous Environments [2.566571621858397]
  We propose Vision-Click-Action (VCA), a framework that replaces textual commands with direct, click-based visual interaction.<n>VCA reduces interpretation errors, lowers cognitive load, and provides a practical and scalable alternative to language-driven interfaces for real-world robotic manipulation.
  arXiv  Detail & Related papers  (2026-02-27T01:23:35Z)
• Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting [15.173629384871013]
  Visual Attentive Prompting (VAP) is a training-free perceptual adapter that equips frozen Vision-Language-Action models with selective attention.<n>VAP consistently outperforms generic policies and token-learning baselines in both success rate and correct-object manipulation.
  arXiv  Detail & Related papers  (2025-12-23T03:13:39Z)
• A Multimodal Depth-Aware Method For Embodied Reference Understanding [56.30142869506262]
  Embodied Reference Understanding requires identifying a target object in a visual scene based on both language instructions and pointing cues.<n>We propose a novel ERU framework that jointly leverages data augmentation, depth-map modality, and a depth-aware decision module.
  arXiv  Detail & Related papers  (2025-10-09T14:32:21Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• Position-Aware Contrastive Alignment for Referring Image Segmentation [65.16214741785633]
  We present a position-aware contrastive alignment network (PCAN) to enhance the alignment of multi-modal features. Our PCAN consists of two modules: 1) Position Aware Module (PAM), which provides position information of all objects related to natural language descriptions, and 2) Contrastive Language Understanding Module (CLUM), which enhances multi-modal alignment.
  arXiv  Detail & Related papers  (2022-12-27T09:13:19Z)
• Bear the Query in Mind: Visual Grounding with Query-conditioned Convolution [26.523051615516742]
  We propose a Query-conditioned Convolution Module (QCM) that extracts query-aware visual features by incorporating query information into the generation of convolutional kernels. Our method achieves state-of-the-art performance on three popular visual grounding datasets.
  arXiv  Detail & Related papers  (2022-06-18T04:26:39Z)
• Multi-modal Transformers Excel at Class-agnostic Object Detection [105.10403103027306]
  We argue that existing methods lack a top-down supervision signal governed by human-understandable semantics. We develop an efficient and flexible MViT architecture using multi-scale feature processing and deformable self-attention. We show the significance of MViT proposals in a diverse range of applications.
  arXiv  Detail & Related papers  (2021-11-22T18:59:29Z)
• Learning to Ground Visual Objects for Visual Dialog [26.21407651331964]
  We propose a novel approach to Learn to Ground visual objects for visual dialog. A posterior distribution over visual objects is inferred from both context (history and questions) and answers. A prior distribution, which is inferred from context only, is used to approximate the posterior distribution so that appropriate visual objects can be grounded even without answers.
  arXiv  Detail & Related papers  (2021-09-13T14:48:44Z)
• Object-and-Action Aware Model for Visual Language Navigation [70.33142095637515]
  Vision-and-Language Navigation (VLN) is unique in that it requires turning relatively general natural-language instructions into robot agent actions. We propose an Object-and-Action Aware Model (OAAM) that processes these two different forms of natural language based instruction separately. This enables each process to match object-centered/action-centered instruction to their own counterpart visual perception/action orientation flexibly.
  arXiv  Detail & Related papers  (2020-07-29T06:32:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.