GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models

• URL: http://arxiv.org/abs/2608.24714v1
• Date: Tue, 25 Aug 2026 15:34:13 GMT
• Title: GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models
• Abstract summary: World-Action Models (WAMs) jointly learn future visual prediction and action generation, using video dynamics as a representation-learning signal for robotic manipulation.<n>We propose textbfGaussianWAM, a training-time representation-enhancement framework that organizes geometric and semantic supervision through a 3D Gaussian field.
• Score: 21.006803571591707
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: World-Action Models (WAMs) jointly learn future visual prediction and action generation, using video dynamics as a representation-learning signal for robotic manipulation. However, their video latents are primarily optimized for visual prediction and are not explicitly encouraged to preserve cross-view geometric structure or spatially localized, object-relevant semantics. We propose \textbf{GaussianWAM}, a training-time representation-enhancement framework that organizes geometric and semantic supervision through a 3D Gaussian field. Given synchronized multi-view observations, frozen geometry and vision foundation models provide depth, camera parameters, and dense semantic features. GaussianWAM binds these heterogeneous signals to shared Gaussian primitives and renders spatially aligned semantic, depth, and coverage targets, which are distilled into the current-observation representations of the WAM. All teacher models, Gaussian components, and auxiliary prediction heads are removed after training, leaving the original WAM inference path without additional modules or forward computation. On LIBERO-Plus, GaussianWAM improves FastWAM from 52.05\% to 71.29\% and Cosmos Policy from 71.52\% to 77.30\%. Direct CLIP and VGGT distillation already establishes a strong FastWAM baseline of 69.37\%, while Gaussian-field unification further improves it to 71.29\%, supporting the benefit of spatially organizing heterogeneous teacher signals. GaussianWAM also improves performance on standard LIBERO and shows positive transfer trends on RoboTwin and real-world manipulation. These results suggest that training-time Gaussian distillation provides a practical way to inject geometry- and semantics-related supervision into WAM representations without changing their deployment architecture.

Related papers

• GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation [55.47130289185285]
  Vision-Language-Action (VLA) policies have advanced language-conditioned robotic manipulation.<n>We present textbfMethodname, a compact, policy-native extension that inserts textbfWorld State Tokens and textbfWorld Prediction Tokens directly into the VLA backbone.
  arXiv  Detail & Related papers  (2026-08-26T11:41:55Z)
• GaussVLA: Geometry-Aware Spatial Reasoning for Vision-Language-Action Model [8.618395711206896]
  GaussVLA encodes visual observations as flat 2D patch tokens that carry no intrinsic geometric structure.<n>It pools geometrically salient regions with learned queries, and emphDepth-Aware Chain-of-Thought (DA-CoT) that performs structured, non-autoregressive geometric reasoning.<n>On LIBERO, it achieves 93.5% average success and 100.0% success on the Spatial suite with only 200M parameters.
  arXiv  Detail & Related papers  (2026-08-25T06:28:28Z)
• Gaussian-JEPA: Joint-Embedding Predictive Learning for 3D Gaussian Splats [91.55220894541151]
  3D Gaussian Splatting (3DGS) represents 3D content with anisotropic primitives that jointly encode geometry and appearance.<n>We introduce Gaussian-JEPA, which predicts representations of held-out Gaussian token blocks from visible context.
  arXiv  Detail & Related papers  (2026-08-16T09:43:50Z)
• Learning Adaptive Semantic Gaussian Allocation for 3D Occupancy [38.71779538292353]
  We show how to select the most useful Gaussians when their total number must be limited to control memory and rendering.<n>We propose the Semantic Gaussian Allocation Transformer (SAGFormer), which uses Gaussian attributes and local geometric-semantic features to score candidates and select a fixed final set.<n> Experiments on nuScenes-SurroundOcc and SSCBench-KITTI-360 show that SAGFormer improves occupancy prediction under the evaluated protocols and yields more semantically consistent and better-utilized Gaussian representations.
  arXiv  Detail & Related papers  (2026-07-24T01:52:49Z)
• Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images [62.31892851760248]
  COVScene is a pose-free semantic Gaussian framework that couples renderable primitives with a dense semantic occupancy field.<n>We show that COVScene maintains competitive rendering quality, improves open-vocabulary segmentation, and achieves stronger semantic occupancy prediction than the self-supervised baseline.
  arXiv  Detail & Related papers  (2026-07-02T03:00:23Z)
• ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understanding [7.610505486431266]
  We introduce ShelfGaussian, an open-vocabulary multi-modal Gaussian-based 3D scene understanding framework supervised by off-the-shelf vision foundation models.<n>Existing methods either model objects as closed-set semantic Gaussians supervised by annotated 3D labels, neglecting their rendering ability, or learn open-set Gaussian representations via purely 2D self-supervision.
  arXiv  Detail & Related papers  (2025-12-03T02:06:09Z)
• AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splatting [46.677120329555486]
  AutoOcc is a vision-centric automated pipeline for semantic occupancy annotation.<n>We formulate the open-ended semantic 3D occupancy reconstruction task to automatically generate scene occupancy.<n>Our framework outperforms existing automated occupancy annotation methods without human labels.
  arXiv  Detail & Related papers  (2025-02-07T14:58:59Z)
• GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding [44.68350305790145]
  GaussTR is a novel Transformer framework that unifies sparse 3D modeling with foundation model alignment through Gaussian representations to advance 3D spatial understanding.<n>Experiments on the Occ3D-nuScenes dataset demonstrate GaussTR's state-of-the-art zero-shot performance of 12.27 mIoU, along with a 40% reduction in training time.<n>These results highlight the efficacy of GaussTR for scalable and holistic 3D spatial understanding, with promising implications in autonomous driving and embodied agents.
  arXiv  Detail & Related papers  (2024-12-17T18:59:46Z)
• MonoGSDF: Exploring Monocular Geometric Cues for Gaussian Splatting-Guided Implicit Surface Reconstruction [86.87464903285208]
  We introduce MonoGSDF, a novel method that couples primitives with a neural Signed Distance Field (SDF) for high-quality reconstruction.<n>To handle arbitrary-scale scenes, we propose a scaling strategy for robust generalization.<n>Experiments on real-world datasets outperforms prior methods while maintaining efficiency.
  arXiv  Detail & Related papers  (2024-11-25T20:07:07Z)
• ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their Self-Supervised Pretraining [112.40071212468843]
  3D Gaussian Splatting (3DGS) has become the de facto method of 3D representation in many vision tasks.<n>We build a large-scale dataset of 3DGS using the commonly used ShapeNet, ModelNet and averse.<n>We introduce Gaussian-MAE, which highlights the unique benefits of representation learning from Gaussian parameters.
  arXiv  Detail & Related papers  (2024-08-20T14:49:14Z)
• GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction [70.65250036489128]
  3D semantic occupancy prediction aims to obtain 3D fine-grained geometry and semantics of the surrounding scene. We propose an object-centric representation to describe 3D scenes with sparse 3D semantic Gaussians. GaussianFormer achieves comparable performance with state-of-the-art methods with only 17.8% - 24.8% of their memory consumption.
  arXiv  Detail & Related papers  (2024-05-27T17:59:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.