Controllable Dynamic 3D Shape Generation via 3D Trajectories and Text

• URL: http://arxiv.org/abs/2606.05162v1
• Date: Wed, 03 Jun 2026 17:58:01 GMT
• Title: Controllable Dynamic 3D Shape Generation via 3D Trajectories and Text
• Abstract summary: T2Mo is a feed-forward framework for controllable dynamic 3D shape generation conditioned on 3D trajectories and text.<n>We propose a shape-grounded trajectory embedding that maps an input trajectory set into a shape-aware token set covering the entire object.
• Score: 33.77648928850077
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce T2Mo, a feed-forward framework for controllable dynamic 3D shape generation conditioned on 3D trajectories and text. Due to the inherent ambiguity of language, generating precisely intended motions using text alone remains challenging. To address this, we adopt 3D trajectories as controllable spatial guidance, specifying the exact paths along which selected points should move. By combining both, T2Mo generates object motions that spatially adhere to the given trajectories while globally reflecting the text semantics. To robustly handle trajectory inputs with arbitrary configurations, ranging from dense to sparse and unevenly distributed, we further propose a shape-grounded trajectory embedding that maps an input trajectory set into a shape-aware token set covering the entire object. We conduct extensive comparisons against text-based baselines and cascaded video-based baselines that combine trajectory-guided video generation with video-to-dynamic mesh generation. Quantitative and qualitative evaluations, along with user studies, demonstrate that our approach produces motions that more faithfully follow the given prompts with higher expressiveness while preserving motion quality.

Related papers

• Follow Your Track: Precise Skeleton Animation Controlled by 3D Trajectories [93.39170866007431]
  4D generation aims to animate 3D objects with realistic motion, holding great promise for applications.<n>Existing methods typically decouple 3D asset generation from motion synthesis.<n>We present textttACT, a trajectory-conditioned framework for topology-general skeletal animation.
  arXiv  Detail & Related papers  (2026-06-24T03:18:10Z)
• LooseControlVideo: Directorial Video Control using Spatial Blocking [44.01478894715323]
  We present LooseControlVideo, a framework that enables intuitive and expressive control by using sparse, oriented 3D boxes as a "blocking" proxy.<n>We achieve this by fine-tuning a Wan 2.2 backbone on a video dataset annotated with DNOCS, a novel encoding for 3D size and orientation.<n>Our findings indicate a 1.2x to 3x improvement in Trajectory Error; 2x improvement in Rigid Motion Consistency; and a 1.5x to 2x increase in Occlusion Accuracy.
  arXiv  Detail & Related papers  (2026-06-17T18:32:13Z)
• VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Prediction [59.303842406260124]
  VGGT-Edit is a feed-forward framework for text-conditioned native 3D scene editing.<n>We introduce depth-synchronized text injection to align semantic guidance with the backbone's spatial poses.<n> VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed.
  arXiv  Detail & Related papers  (2026-05-14T17:59:04Z)
• TrajVG: 3D Trajectory-Coupled Visual Geometry Learning [34.47385438436523]
  TrajVG is a reconstruction framework that makes cross-frame 3D correspondence explicit.<n>We reformulate the same coupling constraints into self-supervised objectives using only pseudo 2D tracks.<n>Experiments across 3D tracking, pose estimation, pointmap reconstruction, and video depth show that TrajVG surpasses the current feedforward performance baseline.
  arXiv  Detail & Related papers  (2026-02-04T11:13:55Z)
• TGT: Text-Grounded Trajectories for Locally Controlled Video Generation [33.989722489622075]
  We introduce Text-Grounded Trajectories (TGT), a framework that conditions video generation on trajectories paired with localized text descriptions.<n>TGT achieves higher visual quality, more accurate text alignment, and improved motion controllability compared with prior approaches.
  arXiv  Detail & Related papers  (2025-10-16T19:45:27Z)
• Drag4D: Align Your Motion with Text-Driven 3D Scene Generation [77.79131321983677]
  Drag4D is an interactive framework that integrates object motion control within text-driven 3D scene generation.<n>This framework enables users to define 3D trajectories for the 3D objects generated from a single image, seamlessly integrating them into a high-quality 3D background.
  arXiv  Detail & Related papers  (2025-09-26T05:23:45Z)
• DiTraj: training-free trajectory control for video diffusion transformer [34.05715460730871]
  Trajectory control represents a user-friendly task in controllable video generation.<n>We propose DiTraj, a training-free framework for trajectory control in text-to-video generation tailored for DiT.<n>Our method outperforms previous methods in both video quality and trajectory controllability.
  arXiv  Detail & Related papers  (2025-09-26T03:53:31Z)
• St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World [106.91539872943864]
  St4RTrack is a framework that simultaneously reconstructs and tracks dynamic video content in a world coordinate frame from RGB inputs.<n>We predict both pointmaps at the same moment, in the same world, capturing both static and dynamic scene geometry.<n>We establish a new extensive benchmark for world-frame reconstruction and tracking, demonstrating the effectiveness and efficiency of our unified, data-driven framework.
  arXiv  Detail & Related papers  (2025-04-17T17:55:58Z)
• Vision-based Discovery of Nonlinear Dynamics for 3D Moving Target [11.102585080028945]
  We propose a vision-based approach to automatically uncover governing equations of nonlinear dynamics for 3D moving targets via raw videos recorded by a set of cameras. This framework is capable of effectively handling the challenges associated with measurement data, e.g., noise in the video, imprecise tracking of the target that causes data missing, etc.
  arXiv  Detail & Related papers  (2024-04-27T11:13:55Z)
• AutoDecoding Latent 3D Diffusion Models [95.7279510847827]
  We present a novel approach to the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and de-normalization operations.
  arXiv  Detail & Related papers  (2023-07-07T17:59:14Z)
• Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding [61.57847727651068]
  Temporal sentence grounding aims to localize a target segment in an untrimmed video semantically according to a given sentence query. Most previous works focus on learning frame-level features of each whole frame in the entire video, and directly match them with the textual information. We propose a novel Motion- and Appearance-guided 3D Semantic Reasoning Network (MA3SRN), which incorporates optical-flow-guided motion-aware, detection-based appearance-aware, and 3D-aware object-level features.
  arXiv  Detail & Related papers  (2022-03-06T13:57:09Z)
• Monocular Quasi-Dense 3D Object Tracking [99.51683944057191]
  A reliable and accurate 3D tracking framework is essential for predicting future locations of surrounding objects and planning the observer's actions in numerous applications such as autonomous driving. We propose a framework that can effectively associate moving objects over time and estimate their full 3D bounding box information from a sequence of 2D images captured on a moving platform.
  arXiv  Detail & Related papers  (2021-03-12T15:30:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.