Velox: Learning Representations of 4D Geometry and Appearance

• URL: http://arxiv.org/abs/2605.04527v1
• Date: Wed, 06 May 2026 06:12:19 GMT
• Title: Velox: Learning Representations of 4D Geometry and Appearance
• Abstract summary: We introduce a framework for learning latent representations of 4D objects which are descriptive, faithfully capturing object geometry and appearance.<n>We evaluate it across three downstream tasks -- video-to-4D generation, 3D tracking, and cloth simulation via image-to-4D generation.
• Score: 37.12487786007913
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We introduce a framework for learning latent representations of 4D objects which are descriptive, faithfully capturing object geometry and appearance; compressive, aiding in downstream efficiency; and accessible, requiring minimal input, i.e., an unstructured dynamic point cloud, to construct. Specifically, Velox trains an encoder to compress spatiotemporal color point clouds into a set of dynamic shape tokens. These tokens are supervised using two complementary decoders: a 4D surface decoder, which models the time-varying surface distribution capturing the geometry; and a Gaussian decoder, which maps the tokens to 3D Gaussians, helping learn appearance. To demonstrate the utility of our representation, we evaluate it across three downstream tasks -- video-to-4D generation, 3D tracking, and cloth simulation via image-to-4D generation -- and observe strong performances in all settings.

Related papers

• VideoTok4D: A 4D-Aware Video Tokenizer for Compact World Representation [21.308618722721388]
  Video tokenizers have emerged as a cornerstone of modern video modeling.<n>We propose VideoTok4D, a novel 4D-aware video tokenizer for compact world representation.
  arXiv  Detail & Related papers  (2026-09-11T14:03:02Z)
• CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes [2.5603340362916938]
  We present CL4D, the first foundational 4D vision encoder that directly operates on dynamic point clouds.<n>We introduce 4DVLM, a 4D vision-language model that conditions language generation on dynamic geometric representations.<n> CL4D achieves state-of-the-art performance, with improvements of approximately 16.75% over prior methods.
  arXiv  Detail & Related papers  (2026-08-19T09:37:15Z)
• Syn4D: A Multiview Synthetic 4D Dataset [92.50132053229817]
  3D reconstruction and tracking of dynamic scenes from monocular video remains an important open challenge in computer vision.<n>To address this limitation, we introduce Syn4D, a multiview synthetic dynamic scenes.<n>Key feature of Syn4D is the ability to unproject any pixel into 3D.
  arXiv  Detail & Related papers  (2026-05-06T17:59:55Z)
• Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models [79.06910348413861]
  We introduce Diff4Splat, a feed-forward method that synthesizes controllable and explicit 4D scenes from a single image.<n>Given a single input image, a camera trajectory, and an optional text prompt, Diff4Splat directly predicts a deformable 3D Gaussian field that encodes appearance, geometry, and motion.
  arXiv  Detail & Related papers  (2025-11-01T11:16:25Z)
• Easi3R: Estimating Disentangled Motion from DUSt3R Without Training [69.51086319339662]
  We introduce Easi3R, a simple yet efficient training-free method for 4D reconstruction.<n>Our approach applies attention adaptation during inference, eliminating the need for from-scratch pre-training or network fine-tuning.<n>Our experiments on real-world dynamic videos demonstrate that our lightweight attention adaptation significantly outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-03-31T17:59:58Z)
• 4D Panoptic Scene Graph Generation [102.22082008976228]
  We introduce 4D Panoptic Scene Graph (PSG-4D), a new representation that bridges the raw visual data perceived in a dynamic 4D world and high-level visual understanding. Specifically, PSG-4D abstracts rich 4D sensory data into nodes, which represent entities with precise location and status information, and edges, which capture the temporal relations. We propose PSG4DFormer, a Transformer-based model that can predict panoptic segmentation masks, track masks along the time axis, and generate the corresponding scene graphs.
  arXiv  Detail & Related papers  (2024-05-16T17:56:55Z)
• Learning Coherent Matrixized Representation in Latent Space for Volumetric 4D Generation [48.671462912294594]
  We propose a framework that generates 4D sequences with dynamic evolution in shape and color across spatial and temporal dimensions.<n>We first use a coherent 3D shape and color modeling to encode shape and color each detailed 3D geometry frame into a latent space.<n>Then we propose a matrixized 4D sequence representation allowing efficient diffusion model operation.
  arXiv  Detail & Related papers  (2024-03-20T01:59:43Z)
• 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency [118.15258850780417]
  We present textbf4DGen, a novel framework for grounded 4D content creation.<n>Our pipeline facilitates controllable 4D generation, enabling users to specify the motion via monocular video or adopt image-to-video generations.<n>Compared to existing video-to-4D baselines, our approach yields superior results in faithfully reconstructing input signals.
  arXiv  Detail & Related papers  (2023-12-28T18:53:39Z)
• Im4D: High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenes [69.52540205439989]
  We introduce Im4D, a hybrid representation that consists of a grid-based geometry representation and a multi-view image-based appearance representation. We represent the scene appearance by the original multi-view videos and a network that learns to predict the color of a 3D point from image features. We show that Im4D state-of-the-art performance in rendering quality and can be trained efficiently, while realizing real-time rendering with a speed of 79.8 FPS for 512x512 images.
  arXiv  Detail & Related papers  (2023-10-12T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.