CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes

• URL: http://arxiv.org/abs/2608.18734v1
• Date: Wed, 19 Aug 2026 09:37:15 GMT
• Title: CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes
• Abstract summary: We present CL4D, the first foundational 4D vision encoder that directly operates on dynamic point clouds.<n>We introduce 4DVLM, a 4D vision-language model that conditions language generation on dynamic geometric representations.<n> CL4D achieves state-of-the-art performance, with improvements of approximately 16.75% over prior methods.
• Score: 2.5603340362916938
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 4D understanding and reasoning is a fundamental capability for embodied AI agents operating in dynamic physical environments. However, existing vision encoders are largely limited to static 2D images or 3D point clouds without temporal modeling, or to 2D videos that lack accurate geometric depth reasoning. Consequently, current approaches fail to jointly capture spatial structure and motion evolution in dynamic scenes. We present CL4D, the first foundational 4D vision encoder that directly operates on dynamic point clouds, trained with a contrastive learning objective to align spatio-temporal geometric representations with natural language descriptions. By learning a shared embedding space between text and 4D scene dynamics, CL4D enables zero-shot motion-to-text and text-to-motion retrieval in dynamic environments and serves as a foundational 4D vision encoder for downstream 4D vision-language tasks. Building on this encoder, we introduce 4DVLM, a 4D vision-language model that conditions language generation on dynamic geometric representations. 4DVLM is the first VLM designed to operate directly on 4D point clouds without relying on 2D images, 2D videos, or static 3D point clouds. We train CL4D and subsequently 4DVLM on a newly constructed dataset termed DynAction4D capturing diverse human motions across varying object interactions and scene environments. Extensive experiments across multiple 4D human action benchmarks demonstrate that CL4D achieves state-of-the-art performance, with improvements of approximately ~16.75% over prior methods. Furthermore, 4DVLM outperforms frontier video VLMs such as Gemini and GPT-5 even when these models are provided with RGB video sequences corresponding to the same scenes represented as 4D point clouds for 4DVLM.

Related papers

• Full-4D: Generating Full-Scope 4D Scenes from a Single-View Video [63.61099683012546]
  We introduce RealMV-4D, a dataset of synchronized multi-view videos.<n>We then train a multi-view video diffusion model driven by a novel fused time(T)-view(V) attention mechanism.<n>Experiments demonstrate that our method outperforms existing approaches in both visual fidelity and consistency.
  arXiv  Detail & Related papers  (2026-05-25T06:59:26Z)
• 4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding [31.082079260882896]
  We present 4DThinker, the first framework that enables vision-language models to "think with 4D"<n>We first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos.<n>We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly verbose textual tokens and 4D latents to ground the model in dynamic visual semantics.
  arXiv  Detail & Related papers  (2026-05-07T10:48:46Z)
• Velox: Learning Representations of 4D Geometry and Appearance [37.12487786007913]
  We introduce a framework for learning latent representations of 4D objects which are descriptive, faithfully capturing object geometry and appearance.<n>We evaluate it across three downstream tasks -- video-to-4D generation, 3D tracking, and cloth simulation via image-to-4D generation.
  arXiv  Detail & Related papers  (2026-05-06T06:12:19Z)
• VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control [83.92729346325163]
  VerseCrafter is a 4D-aware video world model that enables explicit and coherent control over both camera and object dynamics.<n>Our approach is centered on a novel 4D Geometric Control representation, which encodes the world state through a static background point cloud.<n>These 4D controls are rendered into conditioning signals for a pretrained video diffusion model, enabling the generation of high-fidelity, view-consistent videos.
  arXiv  Detail & Related papers  (2026-01-08T17:28:52Z)
• Uni4D: A Unified Self-Supervised Learning Framework for Point Cloud Videos [70.07088203106443]
  Existing methods rely on explicit knowledge to learn motion, resulting in suboptimal representations.<n>Prior Masked Autoentangler (MAE) frameworks struggle to bridge the gap between low-level geometry and high-level dynamics in 4D data.<n>We propose a novel self-disentangled MAE for learning expressive,riminative, and transferable 4D representations.
  arXiv  Detail & Related papers  (2025-04-07T08:47:36Z)
• Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency [49.875459658889355]
  Free4D is a tuning-free framework for 4D scene generation from a single image.<n>Our key insight is to distill pre-trained foundation models for consistent 4D scene representation.<n>The resulting 4D representation enables real-time, controllable rendering.
  arXiv  Detail & Related papers  (2025-03-26T17:59:44Z)
• VG4D: Vision-Language Model Goes 4D Video Recognition [34.98194339741201]
  Vision-Language Models (VLM) pre-trained on web-scale text-image datasets can learn fine-grained visual concepts. We propose the Vision-Language Models Goes 4D (VG4D) framework to transfer VLM knowledge from visual-text pre-trained models to a 4D point cloud network.
  arXiv  Detail & Related papers  (2024-04-17T17:54:49Z)
• Comp4D: LLM-Guided Compositional 4D Scene Generation [65.5810466788355]
  We present Comp4D, a novel framework for Compositional 4D Generation. Unlike conventional methods that generate a singular 4D representation of the entire scene, Comp4D innovatively constructs each 4D object within the scene separately. Our method employs a compositional score distillation technique guided by the pre-defined trajectories.
  arXiv  Detail & Related papers  (2024-03-25T17:55:52Z)
• Text-To-4D Dynamic Scene Generation [111.89517759596345]
  We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions. Our approach uses a 4D dynamic Neural Radiance Field (NeRF), which is optimized for scene appearance, density, and motion consistency. The dynamic video output generated from the provided text can be viewed from any camera location and angle, and can be composited into any 3D environment.
  arXiv  Detail & Related papers  (2023-01-26T18:14:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.