Forecasting Motion in the Wild

• URL: http://arxiv.org/abs/2604.01015v1
• Date: Wed, 01 Apr 2026 15:18:06 GMT
• Title: Forecasting Motion in the Wild
• Abstract summary: We propose dense point trajectories as visual tokens for behavior, a structured mid-level representation that disentangles motion from appearance.<n>To evaluate at scale, we curate 300 hours of unconstrained animal video with robust shot detection and camera-motion compensation.<n>Experiments show that forecasting trajectory tokens achieves category-agnostic, data-efficient prediction, outperforms state-of-the-art baselines, and generalizes to rare species and morphologies.
• Score: 39.32101614367415
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual intelligence requires anticipating the future behavior of agents, yet vision systems lack a general representation for motion and behavior. We propose dense point trajectories as visual tokens for behavior, a structured mid-level representation that disentangles motion from appearance and generalizes across diverse non-rigid agents, such as animals in-the-wild. Building on this abstraction, we design a diffusion transformer that models unordered sets of trajectories and explicitly reasons about occlusion, enabling coherent forecasts of complex motion patterns. To evaluate at scale, we curate 300 hours of unconstrained animal video with robust shot detection and camera-motion compensation. Experiments show that forecasting trajectory tokens achieves category-agnostic, data-efficient prediction, outperforms state-of-the-art baselines, and generalizes to rare species and morphologies, providing a foundation for predictive visual intelligence in the wild.

Related papers

• Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning [37.17021354294023]
  We present a joint visual-trajectory world-action model that simultaneously forecasts future visual states and instrument trajectories from historical surgical observations.<n>We observe progressive visual degradation and accumulated trajectory errors over longer prediction horizons.
  arXiv  Detail & Related papers  (2026-08-20T17:18:02Z)
• LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning [51.969318585152116]
  LatentPilot exploits future observations during training as a valuable data source to learn action-conditioned visual dynamics.<n>We propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent's behavior distribution.<n> Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot's superior understanding of environment-action dynamics in scene.
  arXiv  Detail & Related papers  (2026-03-31T02:21:59Z)
• Integrating Specialized and Generic Agent Motion Prediction with Dynamic Occupancy Grid Maps [3.3894571022475066]
  We propose a unified framework to simultaneously predict future occupancy state grids, vehicle grids, and scene flow grids.<n>Our approach is centered on a tailored, interdependent loss function that captures inter-grid dependencies and enables diverse future predictions.<n> Evaluations on real-world nuScenes and Woven Planet datasets demonstrate superior prediction performances for dynamic vehicles and generic dynamic scene elements.
  arXiv  Detail & Related papers  (2026-02-08T12:13:06Z)
• Generalist Forecasting with Frozen Video Models via Latent Diffusion [35.96406989431198]
  We show a strong correlation between a vision model's perceptual ability and its generalist forecasting performance over short time horizons.<n>Our results highlight the value of bridging representation learning and generative modeling for temporally grounded video understanding.
  arXiv  Detail & Related papers  (2025-07-18T14:14:19Z)
• A polar prediction model for learning to represent visual transformations [10.857320773825357]
  We propose a self-supervised representation-learning framework that exploits the regularities of natural videos to compute accurate predictions. When trained on natural video datasets, our framework achieves better prediction performance than traditional motion compensation. Our framework offers a principled framework for understanding how the visual system represents sensory inputs in a form that simplifies temporal prediction.
  arXiv  Detail & Related papers  (2023-03-06T19:00:59Z)
• PREF: Predictability Regularized Neural Motion Fields [68.60019434498703]
  Knowing 3D motions in a dynamic scene is essential to many vision applications. We leverage a neural motion field for estimating the motion of all points in a multiview setting. We propose to regularize the estimated motion to be predictable.
  arXiv  Detail & Related papers  (2022-09-21T22:32:37Z)
• Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction [63.62263239934777]
  We conduct an indepth study on various pose representations with a focus on their effects on the motion prediction task. We propose a novel RNN architecture termed AHMR (Attentive Hierarchical Motion Recurrent network) for motion prediction. Our approach outperforms the state-of-the-art methods in short-term prediction and achieves much enhanced long-term prediction proficiency.
  arXiv  Detail & Related papers  (2021-12-30T10:45:22Z)
• Learning Dynamics via Graph Neural Networks for Human Pose Estimation and Tracking [98.91894395941766]
  We propose a novel online approach to learning the pose dynamics, which are independent of pose detections in current fame. Specifically, we derive this prediction of dynamics through a graph neural network(GNN) that explicitly accounts for both spatial-temporal and visual information. Experiments on PoseTrack 2017 and PoseTrack 2018 datasets demonstrate that the proposed method achieves results superior to the state of the art on both human pose estimation and tracking tasks.
  arXiv  Detail & Related papers  (2021-06-07T16:36:50Z)
• Local Frequency Domain Transformer Networks for Video Prediction [24.126513851779936]
  Video prediction is of interest not only in anticipating visual changes in the real world but has, above all, emerged as an unsupervised learning rule. This paper proposes a fully differentiable building block that can perform all of those tasks separately while maintaining interpretability.
  arXiv  Detail & Related papers  (2021-05-10T19:48:42Z)
• Implicit Latent Variable Model for Scene-Consistent Motion Forecasting [78.74510891099395]
  In this paper, we aim to learn scene-consistent motion forecasts of complex urban traffic directly from sensor data. We model the scene as an interaction graph and employ powerful graph neural networks to learn a distributed latent representation of the scene.
  arXiv  Detail & Related papers  (2020-07-23T14:31:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.