Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting

• URL: http://arxiv.org/abs/2608.10756v1
• Date: Tue, 11 Aug 2026 10:16:30 GMT
• Title: Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting
• Abstract summary: Embodied mobile manipulation requires language, visual observations, three-dimensional scene structure, and action feasibility to be aligned before execution.<n>We study open-vocabulary target grounding with few-shot manipulation in local household workspaces and present an embodied multimodal grounding framework.<n>A task-driven local Semantic-3DGS serves as a shared interface across active sensing, language-conditioned 3D localization, obstacle-aware scene reasoning, base preparation, and semantic conditioning of the action model.
• Score: 4.473350810262674
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied mobile manipulation requires language, visual observations, three-dimensional scene structure, and action feasibility to be aligned before execution. We study open-vocabulary target grounding with few-shot manipulation in local household workspaces and present an embodied multimodal grounding framework that integrates active multi-view Semantic 3D Gaussian Splatting (Semantic-3DGS), reachability-aware base positioning, and a diffusion-based vision-language-action policy. A task-driven local Semantic-3DGS serves as a shared interface across active sensing, language-conditioned 3D localization, obstacle-aware scene reasoning, base preparation, and semantic conditioning of the action model. To preserve pretrained action priors, the 3D semantic cues are injected only into the late action-expert blocks. In expanded 50-trial real-robot evaluations against representative vision-language-action (VLA) approaches, the full system achieves 60% long-horizon success compared with 40% for PointVLA and 28% for DexVLA, and reaches 74% success in heavily cluttered manipulation compared with 52% for the single-view variant and 46% for PointVLA. It also maintains 75% success under a 75 cm height shift and eliminates photo-induced false grasps. These results indicate that explicit, refreshable 3D semantic grounding can improve robustness under clutter, occlusion, viewpoint variation, and embodiment constraints.

Related papers

• VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation [7.753769810587787]
  Vision-Language-Action (VLA) models have emerged as a powerful end-to-end paradigm for robotic manipulation by mapping language instructions and 2D visual inputs directly to actions.<n>These models lack an explicit, scene-level 3D representation, limiting their ability to reason over spatial layouts and geometric constraints.<n>We present VistaVLA, a novel framework that constructs a geometry- and semantics-aware 3D cognitive representation from 3D Gaussian primitives.
  arXiv  Detail & Related papers  (2026-07-14T05:08:50Z)
• Direct Action-Head Injection of A Grounded 3D Point Unlocks Spatial and Task Generalization [6.3170928681089995]
  Vision-Language-Action (VLA) models leverage large-scale vision-language pretraining for flexible robot manipulation.<n>They remain brittle along two axes: spatial generalization, when object positions differ from those seen during training, and task generalization, when a familiar scene is paired with a different language instruction.<n>We propose a lightweight, model-agnostic module that represents the grounding signal in 3D, computes its relative displacement to the gripper, and injects the resulting embedding directly into the action head.
  arXiv  Detail & Related papers  (2026-06-26T02:44:27Z)
• Sparse2Act: Learning Action-Aligned Sparse 3D Representations for Cross-Domain Robot Manipulation [52.00134393320209]
  We introduce Sparse2Act, an observation-action alignment framework for pretraining sparse point-cloud encoders.<n>Masked sparse 3D tokens are trained to organize scene features around the workspace motion paired with the observation.<n>On the LIBERO-10 benchmark, our method achieves 86.9% average success after 500 fine-tuning steps.
  arXiv  Detail & Related papers  (2026-06-10T23:56:01Z)
• GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation [49.16739604572808]
  Vision-Language-Action (VLA) models achieve strong benchmark performance but struggle in real-world deployment with unseen objects.<n>We argue that this stems from the lack of a unified geometry-aware manipulation representation.<n>We propose GEAR-VLA, a VLA framework for learning unified geometry-aware action representations for generalizable robotic manipulation.
  arXiv  Detail & Related papers  (2026-06-07T09:23:16Z)
• PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction [57.63073414949329]
  We propose PointACT, a dual-system 3D-aware VLA policy that integrates hierarchical 3D point cloud representations directly into the action decoding process.<n>PointACT employs a multi-scale point-action interaction mechanism with efficient bottleneck window self-attention, enabling evolving action tokens to densely attend to both local geometric detail and global scene structure.
  arXiv  Detail & Related papers  (2026-05-20T17:10:31Z)
• Universal Pose Pretraining for Generalizable Vision-Language-Action Policies [83.39008378156647]
  Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency.<n>We propose Pose-VLA, a decoupled paradigm that separates VLA training into a pre-training phase for extracting universal 3D spatial priors.<n>Our framework follows a two-stage pre-training pipeline, establishing fundamental spatial grounding via poses followed by motion alignment.
  arXiv  Detail & Related papers  (2026-02-23T11:00:08Z)
• ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation [52.94334113271359]
  ActiveVLA is a vision-language-action framework that empowers robots with active perception capabilities for high-precision, fine-grained manipulation.<n>We show that ActiveVLA achieves precise 3D manipulation and outperforms state-of-the-art baselines on three simulation benchmarks.
  arXiv  Detail & Related papers  (2026-01-13T08:29:07Z)
• DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoning [35.44151923549777]
  Vision-Language-Action (VLA) models have recently shown impressive generalization and language-guided manipulation capabilities.<n>Their performance degrades on tasks requiring precise spatial reasoning due to limited spatial reasoning inherited from Vision-Language Models (VLMs)<n>We present DepthVLA, a simple yet effective VLA architecture that explicitly incorporates spatial awareness through a pretrained depth prediction module.
  arXiv  Detail & Related papers  (2025-10-15T10:09:00Z)
• ReasonGrounder: LVLM-Guided Hierarchical Feature Splatting for Open-Vocabulary 3D Visual Grounding and Reasoning [68.4209681278336]
  Open-vocabulary 3D visual grounding and reasoning aim to localize objects in a scene based on implicit language descriptions.<n>Current methods struggle because they rely heavily on fine-tuning with 3D annotations and mask proposals.<n>We propose ReasonGrounder, an LVLM-guided framework that uses hierarchical 3D feature Gaussian fields for adaptive grouping.
  arXiv  Detail & Related papers  (2025-03-30T03:40:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.