MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

• URL: http://arxiv.org/abs/2607.16192v1
• Date: Fri, 17 Jul 2026 17:59:38 GMT
• Title: MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction
• Abstract summary: We study how to learn this anticipation from ordinary monocular videos of human-object interaction.<n>MotionForesight predicts future 3D trajectories for points on the manipulated object.<n>We start from a dense 3D tracker built on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames.
• Score: 12.549249041637976
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. We study how to learn this anticipation from ordinary monocular videos of human-object interaction. Given a short observed video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion forecasting without any assumptions on the object properties. Our key insight is that video prediction models already encode rich priors about how objects move during human interactions. We redirect these priors from pixel prediction toward future 3D scene flow. We start from a dense 3D tracker built on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames. We replace future RGB and geometry with learned mask latents and train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large video and tracking components. Using just 40k human videos and no auxiliary inputs such as language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training videos. These results show that we can efficiently re-purpose video priors into explicit geometric forecasts for embodied intelligence. https://motionforesight.github.io/

Related papers

• ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos [48.24897274501108]
  We introduce a 3D object-centric dynamics model that predicts future 6-DoF poses and trajectories of rigid objects from short egocentric video sequences.<n>Unlike conventional world or dynamics models that operate in pixel or latent space, ObjectForesight represents the world explicitly in 3D at the object level.<n>We show that ObjectForesight achieves significant gains in accuracy, geometric consistency, and generalization to unseen objects and scenes.
  arXiv  Detail & Related papers  (2026-01-08T18:58:08Z)
• FIction: 4D Future Interaction Prediction from Video [63.37136159797888]
  We introduce FIction for 4D future interaction prediction from videos.<n>Given an input video of a human activity, the goal is to predict which objects at what 3D locations the person will interact with in the next time period.
  arXiv  Detail & Related papers  (2024-12-01T18:44:17Z)
• Predicting Long-horizon Futures by Conditioning on Geometry and Time [49.86180975196375]
  We explore the task of generating future sensor observations conditioned on the past. We leverage the large-scale pretraining of image diffusion models which can handle multi-modality. We create a benchmark for video prediction on a diverse set of videos spanning indoor and outdoor scenes.
  arXiv  Detail & Related papers  (2024-04-17T16:56:31Z)
• 3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes [68.66237114509264]
  We present a framework capable of learning 3D-grounded visual intuitive physics models from videos of complex scenes with fluids. We show our model can make long-horizon future predictions by learning from raw images and significantly outperforms models that do not employ an explicit 3D representation space.
  arXiv  Detail & Related papers  (2023-04-22T19:28:49Z)
• T3VIP: Transformation-based 3D Video Prediction [49.178585201673364]
  We propose a 3D video prediction (T3VIP) approach that explicitly models the 3D motion by decomposing a scene into its object parts. Our model is fully unsupervised, captures the nature of the real world, and the observational cues in image and point cloud domains constitute its learning signals. To the best of our knowledge, our model is the first generative model that provides an RGB-D video prediction of the future for a static camera.
  arXiv  Detail & Related papers  (2022-09-19T15:01:09Z)
• We are More than Our Joints: Predicting how 3D Bodies Move [63.34072043909123]
  We train a novel variational autoencoder that generates motions from latent frequencies. Experiments show that our method produces state-of-the-art results and realistic 3D body animations.
  arXiv  Detail & Related papers  (2020-12-01T16:41:04Z)
• 3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators [24.181604511269096]
  We propose an action-conditioned dynamics model that predicts scene changes caused by object and agent interactions in a viewpoint-in 3D neural scene representation space. In this space, objects do not interfere with one another and their appearance persists over time and across viewpoints. We show our model generalizes well its predictions across varying number and appearances of interacting objects as well as across camera viewpoints.
  arXiv  Detail & Related papers  (2020-11-12T16:15:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.