MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction

• URL: http://arxiv.org/abs/2606.18558v1
• Date: Wed, 17 Jun 2026 00:19:00 GMT
• Title: MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction
• Abstract summary: MolmoMotion is a general motion forecasting model that supports both autoregressive coordinate prediction and flow-matching-based trajectory generation.<n>It accurately predicts diverse motion patterns with different language instructions, and significantly outperforms existing motion prediction baselines on PointMotionBench.<n>It improves training efficiency and generalization for robot manipulation, and its predicted trajectories provide effective motion guidance for generative models to synthesize videos with more realistic object motion.
• Score: 48.93378286720386
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Motion forecasting is central to visual intelligence: agents must anticipate how objects will move in order to plan actions, reason about physical interactions, and synthesize realistic futures. We argue that 3D points in world coordinates provide a general representation that is class-agnostic, view-stable, compact, and directly useful for downstream tasks. We formalize the task of goal-conditioned 3D point motion forecasting: given a short visual history, a set of 3D query points on an object of interest, and a language description of the intended goal, the model predicts the future 3D trajectory of each point. We introduce a full stack to study this task at scale: (1) MolmoMotion-1M is a large corpus of action-described, object-grounded 3D point trajectories annotated from 1.16M unconstrained videos; (2) PointMotionBench is a human-verified benchmark spanning 111 object categories and 61 motion types; and (3) MolmoMotion is a general motion forecasting model that supports both autoregressive coordinate prediction and flow-matching-based trajectory generation. MolmoMotion accurately predicts diverse motion patterns with different language instructions, and significantly outperforms existing motion prediction baselines on PointMotionBench. Finally, we show that the learned 3D motion prior transfers well to downstream applications: it improves training efficiency and generalization for robot manipulation, and its predicted trajectories provide effective motion guidance for generative models to synthesize videos with more realistic object motion.

Related papers

• ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos [48.24897274501108]
  We introduce a 3D object-centric dynamics model that predicts future 6-DoF poses and trajectories of rigid objects from short egocentric video sequences.<n>Unlike conventional world or dynamics models that operate in pixel or latent space, ObjectForesight represents the world explicitly in 3D at the object level.<n>We show that ObjectForesight achieves significant gains in accuracy, geometric consistency, and generalization to unseen objects and scenes.
  arXiv  Detail & Related papers  (2026-01-08T18:58:08Z)
• Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos [24.111891848073288]
  Embodied world models aim to predict and interact with the physical world through visual observations and actions.<n>MTV-World introduces Multi-view Trajectory-Video control for precise visuomotor prediction.<n>MTV-World achieves precise control execution and accurate physical interaction modeling in complex dual-arm scenarios.
  arXiv  Detail & Related papers  (2025-11-17T02:17:04Z)
• ManiTrend: Bridging Future Generation and Action Prediction with 3D Flow for Robotic Manipulation [11.233768932957771]
  3D flow represents the motion trend of 3D particles within a scene.<n>ManiTrend is a unified framework that models the dynamics of 3D particles, vision observations and manipulation actions.<n>Our method achieves state-of-the-art performance with high efficiency.
  arXiv  Detail & Related papers  (2025-02-14T09:13:57Z)
• Articulate That Object Part (ATOP): 3D Part Articulation via Text and Motion Personalization [12.944411575346528]
  ATOP (Articulate That Object Part) is a novel few-shot method based on motion personalization to articulate a static 3D object with respect to a part and its motion as prescribed in a text prompt.<n>We show that our method can generate realistic motion samples with higher accuracy, leading to more generalizable 3D motion predictions.
  arXiv  Detail & Related papers  (2025-02-11T05:47:16Z)
• M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenes [66.44171200767839]
  M3Bench is a new benchmark for whole-body motion generation in mobile manipulation tasks.<n>M3Bench features 30,000 object rearrangement tasks across 119 diverse scenes.<n>M3Bench and M3BenchMaker aim to advance robotics research toward more adaptive and capable mobile manipulation.
  arXiv  Detail & Related papers  (2024-10-09T08:38:21Z)
• DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos [76.01906393673897]
  We propose a self-supervised method to jointly learn 3D motion and depth from monocular videos. Our system contains a depth estimation module to predict depth, and a new decomposed object-wise 3D motion (DO3D) estimation module to predict ego-motion and 3D object motion. Our model delivers superior performance in all evaluated settings.
  arXiv  Detail & Related papers  (2024-03-09T12:22:46Z)
• MotionTrack: Learning Motion Predictor for Multiple Object Tracking [68.68339102749358]
  We introduce a novel motion-based tracker, MotionTrack, centered around a learnable motion predictor. Our experimental results demonstrate that MotionTrack yields state-of-the-art performance on datasets such as Dancetrack and SportsMOT.
  arXiv  Detail & Related papers  (2023-06-05T04:24:11Z)
• Semi-Weakly Supervised Object Kinematic Motion Prediction [56.282759127180306]
  Given a 3D object, kinematic motion prediction aims to identify the mobile parts as well as the corresponding motion parameters. We propose a graph neural network to learn the map between hierarchical part-level segmentation and mobile parts parameters. The network predictions yield a large scale of 3D objects with pseudo labeled mobility information.
  arXiv  Detail & Related papers  (2023-03-31T02:37:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.