GST-VLA: Structured Gaussian Spatial Tokens for 3D Depth-Aware Vision-Language-Action Models

• URL: http://arxiv.org/abs/2603.09079v1
• Date: Tue, 10 Mar 2026 01:39:38 GMT
• Title: GST-VLA: Structured Gaussian Spatial Tokens for 3D Depth-Aware Vision-Language-Action Models
• Abstract summary: VLA models encode visual observations as 2D patch tokens with no intrinsic geometric structure.<n>We introduce GST-VLA with two contributions.<n>First, the Gaussian Spatial Tokenizer ( GST) converts frozen dense depth and frozen semantic patch features into 3D primitives.<n>Second, 3D Depth-Aware Chain-of-Thought (DA-CoT) reasoning supervises four structured intermediate spatial thoughts.
• Score: 6.2676602262188625
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: VLA models encode visual observations as 2D patch tokens with no intrinsic geometric structure. We introduce GST-VLA with two contributions. First, the Gaussian Spatial Tokenizer (GST) converts frozen dense depth and frozen semantic patch features into $N_g{=}128$ anisotropic 3D Gaussian primitives, each parameterized by a metric residual mean $μ\in \mathbb{R}^3$, log-scale covariance $\log σ\in \mathbb{R}^3$, and learned opacity $α\in (0,1)$. The covariance eigenstructure encodes local surface orientation, and opacity provides per-primitive geometric confidence, both inaccessible from scalar depth. Spatial attention pooling with learned queries concentrates the fixed token budget on geometrically salient regions rather than distributing uniformly. Second, 3D Depth-Aware Chain-of-Thought (DA-CoT) reasoning supervises four structured intermediate spatial thoughts, covering 3D object grounding, grasp affordance contact geometry, pairwise metric distances, and coarse SE(3) waypoints, as explicit generation targets in the training loss. A cross-attention sublayer at every VLM transformer block provides direct access to the raw 256-primitive Gaussian field during DA-CoT generation. A 300M-parameter flow-matching action expert with mixture-of-experts feedforward sublayers decodes 7-DoF delta action chunks via conditional ODE integration, conditioned on both VLM hidden states and DA-CoT outputs through dual cross-attention. Trained with composite $\mathcal{L}_\mathrm{flow} + \mathcal{L}_\mathrm{CoT} + \mathcal{L}_\mathrm{depth}$ across three progressive stages, GST-VLA achieves 96.4% on LIBERO (+2.0%), and 80.2% on SimplerEnv (+5.4%). Ablations isolate the contribution of each GST component, each DA-CoT thought, and each training stage, confirming independent and synergistic gains concentrated on precision demanding tasks.

Related papers

• SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation [51.46828526392219]
  Reasoning segmentation converts an implicit linguistic conclusion into a precise mask.<n>Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context.<n>We present SeGDeP, an explicit what-where interface.
  arXiv  Detail & Related papers  (2026-09-08T15:11:32Z)
• GaussVLA: Geometry-Aware Spatial Reasoning for Vision-Language-Action Model [8.618395711206896]
  GaussVLA encodes visual observations as flat 2D patch tokens that carry no intrinsic geometric structure.<n>It pools geometrically salient regions with learned queries, and emphDepth-Aware Chain-of-Thought (DA-CoT) that performs structured, non-autoregressive geometric reasoning.<n>On LIBERO, it achieves 93.5% average success and 100.0% success on the Spatial suite with only 200M parameters.
  arXiv  Detail & Related papers  (2026-08-25T06:28:28Z)
• Gaussian-JEPA: Joint-Embedding Predictive Learning for 3D Gaussian Splats [91.55220894541151]
  3D Gaussian Splatting (3DGS) represents 3D content with anisotropic primitives that jointly encode geometry and appearance.<n>We introduce Gaussian-JEPA, which predicts representations of held-out Gaussian token blocks from visible context.
  arXiv  Detail & Related papers  (2026-08-16T09:43:50Z)
• The Lie We Tell: Correcting the Euclidean Fallacy in Vision Language Action Policies via Score Matching on Tangent Space [19.950575615573978]
  We introduce a diffusion framework operating intrinsically on SE(3).<n>Our method injects noise through left-invariant SDEs, predicts scores in the space tangent, and retracts samples via the exponential map.<n>On CALVIN ABC$rightarrow$D, LDA improves average task length from $3.27$ to $3.51$ ($+7.3%$)
  arXiv  Detail & Related papers  (2026-06-01T07:59:29Z)
• VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Prediction [37.542568613707296]
  VGGT-Occ is a framework that embeds geometric tokens throughout the entire pipeline.<n>We introduce Projection-Aware Deformable Attention (PA-DA) to inject geometry into all attention stages.<n>Features are then integrated through a view-quality semantic gate for cross-view consistency.
  arXiv  Detail & Related papers  (2026-05-16T09:51:04Z)
• PointGS: Semantic-Consistent Unsupervised 3D Point Cloud Segmentation with 3D Gaussian Splatting [15.208505879476418]
  Unsupervised point cloud segmentation is critical for embodied artificial intelligence and autonomous driving.<n>This paper proposes PointGS, a simple yet effective pipeline for unsupervised 3D point cloud segmentation.<n> Experiments demonstrate that PointGS outperforms state-of-the-art unsupervised methods.
  arXiv  Detail & Related papers  (2026-05-12T04:44:50Z)
• Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting [78.96382179408054]
  We introduce bfLeGS, a framework that reformulates density control as a parameterized policy network optimized via Reinforcement Learning (RL)<n>To maintain computational tractability, we derive a closed-form solution that reduces the complexity of reward calculation from $O(N2)$ to $O(N)$.<n>Experiments on the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets demonstrate that bfLeGS significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-01T05:04:49Z)
• Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales [0.0]
  We show that transformer training trajectories evolve within only a few coherent directions.<n>In companion work, the same spectral geometry provides early-warning signals of grokking.
  arXiv  Detail & Related papers  (2026-03-14T04:46:05Z)
• 3DGS$^2$-TR: Scalable Second-Order Trust-Region Method for 3D Gaussian Splatting [25.277141070184083]
  We propose 3DGS$2$-TR, a second-order computation for accelerating the training problem in 3D Gaussian Splatting (3DGS)<n>Our method approximates curvature using only the diagonal of the Hessian matrix, efficiently via Hutchinson's method.<n>3DGS$2$-TR is able to achieve better reconstruction quality on standard datasets, using 50% fewer training iterations compared to ADAM and 85% less than 3DGS-LM.
  arXiv  Detail & Related papers  (2026-01-30T23:14:09Z)
• Robust inverse material design with physical guarantees using the Voigt-Reuss Net [0.0]
  We propose a spectrally normalized surrogate for forward and inverse mechanical homogenization with hard physical guarantees.<n>In 3D linear elasticity on an open dataset of biphasic microstructures, a fully connected Voigt-Reuss net trained on $>!7.5times 105$ FFT-based labels with 236 isotropy-in descriptors.<n>Overall, the Voigt-Reuss net unifies accurate, physically admissible forward prediction with large-batch, constraint-consistent inverse design.
  arXiv  Detail & Related papers  (2025-11-14T15:17:37Z)
• Arithmetic-Mean $μ$P for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets [9.94514344279733]
  Arithmetic-Mean $mu$P constrains not each individual layer but the network-wide average one-step pre-activation second moment to a constant scale.<n>We prove that, for one- and two-dimensional convolutional networks, the maximal-update learning rate satisfies $etastar(L)propto L-3/2$; with zero padding, boundary effects are constant-level as $Ngg k$.
  arXiv  Detail & Related papers  (2025-10-05T19:22:50Z)
• econSG: Efficient and Multi-view Consistent Open-Vocabulary 3D Semantic Gaussians [56.85804719947]
  We propose econSG for open-vocabulary semantic segmentation with 3DGS.<n>Our econSG shows state-of-the-art performance on four benchmark datasets compared to the existing methods.
  arXiv  Detail & Related papers  (2025-04-08T13:12:31Z)
• GaussianUDF: Inferring Unsigned Distance Functions through 3D Gaussian Splatting [49.60513072330759]
  We propose a novel approach to bridge the gap between 3D Gaussians and UDFs.<n>Our key idea is to overfit thin and flat 2D Gaussian planes on surfaces, and then, leverage the self-supervision and gradient-based inference.<n>We show our advantages in terms of accuracy, efficiency, completeness, and sharpness of reconstructed open surfaces with boundaries.
  arXiv  Detail & Related papers  (2025-03-25T08:46:55Z)
• Self-Ensembling Gaussian Splatting for Few-Shot Novel View Synthesis [55.561961365113554]
  3D Gaussian Splatting (3DGS) has demonstrated remarkable effectiveness in novel view synthesis (NVS)<n>In this paper, we introduce Self-Ensembling Gaussian Splatting (SE-GS)<n>We achieve self-ensembling by incorporating an uncertainty-aware perturbation strategy during training.<n> Experimental results on the LLFF, Mip-NeRF360, DTU, and MVImgNet datasets demonstrate that our approach enhances NVS quality under few-shot training conditions.
  arXiv  Detail & Related papers  (2024-10-31T18:43:48Z)
• 3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting [58.95801720309658]
  In this paper, we present an implicit surface reconstruction method with 3D Gaussian Splatting (3DGS), namely 3DGSR.<n>The key insight is incorporating an implicit signed distance field (SDF) within 3D Gaussians to enable them to be aligned and jointly optimized.<n>Our experimental results demonstrate that our 3DGSR method enables high-quality 3D surface reconstruction while preserving the efficiency and rendering quality of 3DGS.
  arXiv  Detail & Related papers  (2024-03-30T16:35:38Z)
• Volumetric Attribute Compression for 3D Point Clouds using Feedforward Network with Geometric Attention [36.41214415449853]
  We propose a feedforward linear network that implements higher-order B-spline bases spanning function spaces without eigendecomposition. We show that the number of layers in the normalization at the encoder is equivalent to the number of terms in an inverse Taylor series.
  arXiv  Detail & Related papers  (2023-04-01T15:24:12Z)
• Training \beta-VAE by Aggregating a Learned Gaussian Posterior with a Decoupled Decoder [0.553073476964056]
  Current practices in VAE training often result in a trade-off between the reconstruction fidelity and the continuity$/$disentanglement of the latent space. We present intuitions and a careful analysis of the antagonistic mechanism of the two losses, and propose a simple yet effective two-stage method for training a VAE. We evaluate the method using a medical dataset intended for 3D skull reconstruction and shape completion, and the results indicate promising generative capabilities of the VAE trained using the proposed method.
  arXiv  Detail & Related papers  (2022-09-29T13:49:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.