VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models

• URL: http://arxiv.org/abs/2605.10485v1
• Date: Mon, 11 May 2026 12:44:26 GMT
• Title: VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models
• Abstract summary: Visual backbones of current vision-language-action (VLA) models are predominantly pretrained on 2D image data without explicit 3D geometric supervision.<n>Existing implicit grounding methods partially address this by aligning VLA features with those of 3D-aware foundation models.<n>We propose VEGA, a framework that directly aligns the output of the VLA's visual encoder with spatially-aware features from DINOv2-FiT3D.
• Score: 55.12929235609365
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Precise spatial reasoning is fundamental to robotic manipulation, yet the visual backbones of current vision-language-action (VLA) models are predominantly pretrained on 2D image data without explicit 3D geometric supervision, resulting in representations that lack accurate spatial awareness. Existing implicit spatial grounding methods partially address this by aligning VLA features with those of 3D-aware foundation models, but they rely on empirical layer search and perform alignment on LLM-level visual tokens where spatial structure has already been entangled with linguistic semantics, limiting both generalizability and geometric interpretability. We propose VEGA (Visual Encoder Grounding Alignment), a simple yet effective framework that directly aligns the output of the VLA's visual encoder with spatially-aware features from DINOv2-FiT3D, a DINOv2 model fine-tuned with multi-view consistent 3D Gaussian Splatting supervision. By performing alignment at the visual encoder output level, VEGA grounds spatial awareness before any linguistic entanglement occurs, offering a more interpretable and principled alignment target. The alignment is implemented via a lightweight projector trained with a cosine similarity loss alongside the standard action prediction objective, and is discarded at inference time, introducing no additional computational overhead. Extensive experiments on simulation benchmark and real-world manipulation tasks demonstrate that VEGA consistently outperforms existing implicit spatial grounding baselines, establishing a new state-of-the-art among implicit spatial grounding methods for VLA models.

Related papers

• VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation [7.753769810587787]
  Vision-Language-Action (VLA) models have emerged as a powerful end-to-end paradigm for robotic manipulation by mapping language instructions and 2D visual inputs directly to actions.<n>These models lack an explicit, scene-level 3D representation, limiting their ability to reason over spatial layouts and geometric constraints.<n>We present VistaVLA, a novel framework that constructs a geometry- and semantics-aware 3D cognitive representation from 3D Gaussian primitives.
  arXiv  Detail & Related papers  (2026-07-14T05:08:50Z)
• Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulation [76.16065370700488]
  We introduce Lift3D-VLA, a unified VLA framework that equips models with explicit 3D point cloud reasoning and enables temporally coherent action generation.<n>We show that Lift3D-VLA achieves 10.8% and 11.1% higher mean success rates on MetaWorld and RLBench than the best-performing prior VLA methods.
  arXiv  Detail & Related papers  (2026-07-07T17:59:47Z)
• SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video [88.79625979053873]
  We propose a novel framework, SpaceEra, for pre-trained vision-language models (VLMs)<n>We extend the original framework into a comprehensive system, termed SpaceEra++, which spans data construction, model design, training optimization, and prompting inference.<n>In addition, to enhance spatial reasoning, we develop SpaceAlign, which enforces pairwise object constraints by jointly exploiting absolute coordinates and relative spatial relations.
  arXiv  Detail & Related papers  (2026-07-02T06:56:29Z)
• SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision [12.336413790708738]
  spatial intelligence of large language models (MLLMs) is crucial for understanding and interacting with the 3D world.<n>We propose SpatialSV, a framework designed to internalize robust 3D spatial awareness within MLLMs while simultaneously offering inherent interpretability.
  arXiv  Detail & Related papers  (2026-06-18T08:09:32Z)
• GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigation [75.85672467847631]
  We introduce the Geometry-Aware BEV (GA-BEV) - a compact, 3D-grounded feature representation.<n>We construct BEV spatial maps from RGB-D inputs by projecting visual features into 3D space.<n>We incorporate features from a pretrained 3D foundation model into the BEV space, injecting structural priors learned from large-scale 3D reconstruction tasks.
  arXiv  Detail & Related papers  (2026-05-21T06:20:17Z)
• Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models [9.722829662835233]
  We show that modern Vision-Language Models (VLMs) possess a latent topological map of 3D scenes.<n>By isolating this spatial subspace through cross-scene linear feature extraction, we extract a clean spatial subspace that causally controls the model's spatial outputs.<n>We mathematically shape this latent representation and prove its correspondence to the Laplacian eigenmaps of the scene's 3D Gaussian- kernel graph.
  arXiv  Detail & Related papers  (2026-05-08T02:32:27Z)
• SpatialFly: Geometry-Guided Representation Alignment for UAV Vision-and-Language Navigation in Urban Environments [49.966170814478915]
  We propose a geometry-guided spatial representation framework for UAV VLN.<n>Running on RGB observations without explicit 3D reconstruction, SpatialFly introduces a geometry-guided 2D representation alignment mechanism.<n> Experimental results show that SpatialFly consistently outperforms state-of-the-art UAV VLN baselines across both seen and unseen environments.
  arXiv  Detail & Related papers  (2026-03-22T03:56:58Z)
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding [50.098085774845195]
  We propose a paradigm shift by leveraging the implicit spatial prior within large-scale generation models.<n>We introduce GeneEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator.
  arXiv  Detail & Related papers  (2026-03-19T17:59:58Z)
• Universal Pose Pretraining for Generalizable Vision-Language-Action Policies [83.39008378156647]
  Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency.<n>We propose Pose-VLA, a decoupled paradigm that separates VLA training into a pre-training phase for extracting universal 3D spatial priors.<n>Our framework follows a two-stage pre-training pipeline, establishing fundamental spatial grounding via poses followed by motion alignment.
  arXiv  Detail & Related papers  (2026-02-23T11:00:08Z)
• Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation [34.44214123004662]
  We propose VLM3D, a framework for differentiable semantic and spatial critics.<n>Our core contribution is a dual-language critic signal derived from the VLM's Yes or No log-odds.<n>VLM3D establishes a principled and general path to inject the VLM's rich, language-grounded understanding of both semantics and space into diverse 3D generative pipelines.
  arXiv  Detail & Related papers  (2025-11-18T09:05:26Z)
• From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors [54.84863164684646]
  Existing vision-language-action (VLA) models act in 3D real-world but are typically built on 2D encoders.<n>In this work, we introduce FALCON, a novel paradigm that injects rich 3D spatial tokens into the action head.
  arXiv  Detail & Related papers  (2025-10-20T11:26:45Z)
• Evo-0: Vision-Language-Action Model with Implicit Spatial Understanding [11.222744122842023]
  We introduce a plug-and-play module that implicitly incorporates 3D geometry features into Vision-Language-Action (VLA) models.<n>Our method significantly improves the performance of state-of-the-art VLA models across diverse scenarios.
  arXiv  Detail & Related papers  (2025-07-01T04:05:47Z)
• Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence [13.168559963356952]
  We present Spatial-MLLM, a novel framework for visual-based spatial reasoning from purely 2D observations.<n>Our key insight is to unleash the strong structure prior to the feed-forward visual geometry foundation model.<n>A connector then integrates both features into unified visual tokens for enhanced spatial understanding.
  arXiv  Detail & Related papers  (2025-05-29T17:59:04Z)
• LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models [57.92316645992816]
  Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space.<n>We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs)<n>We demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve their reasoning performance.
  arXiv  Detail & Related papers  (2024-12-03T06:15:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.