ViPro-2: Unsupervised State Estimation via Integrated Dynamics for Guiding Video Prediction

• URL: http://arxiv.org/abs/2508.06335v1
• Date: Fri, 08 Aug 2025 14:10:26 GMT
• Title: ViPro-2: Unsupervised State Estimation via Integrated Dynamics for Guiding Video Prediction
• Abstract summary: We show that ViPro can infer states from observations without providing a full ground truth state in the beginning.<n>We extend the original Orbits dataset with a 3D variant to close the gap to real world scenarios.
• Score: 5.931665251123055
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Predicting future video frames is a challenging task with many downstream applications. Previous work has shown that procedural knowledge enables deep models for complex dynamical settings, however their model ViPro assumed a given ground truth initial symbolic state. We show that this approach led to the model learning a shortcut that does not actually connect the observed environment with the predicted symbolic state, resulting in the inability to estimate states given an observation if previous states are noisy. In this work, we add several improvements to ViPro that enables the model to correctly infer states from observations without providing a full ground truth state in the beginning. We show that this is possible in an unsupervised manner, and extend the original Orbits dataset with a 3D variant to close the gap to real world scenarios.

Related papers

• Can Video World Models Track Unobserved World States? [40.869875403179535]
  Video world models are increasingly used as simulators, yet visual fidelity alone does not show that a model maintains the hidden state of the world.<n>We examine this gap with an action-conditioned video Shell Game, a visual analog of $S_5$ state tracking.<n>We find two mechanisms that do extrapolate, and both carry a state across chunks and revise it in place.
  arXiv  Detail & Related papers  (2026-08-31T12:29:24Z)
• FoMoVLA: Bridging Visual Foresight and Motion Guidance for Vision-Language-Action Models [21.446753234768824]
  Existing visual methods predict future visual states but lack explicit motion guidance.<n>We propose FoMoVLA, a framework that augments VLA representations with explicit foresight-temporal supervision.
  arXiv  Detail & Related papers  (2026-07-16T09:04:50Z)
• Sensorimotor World Models: Perception for Action via Inverse Dynamics [59.121736411156384]
  We introduce a sensorimotor world model trained end-to-end with inverse dynamics regularization.<n>It prevents representation collapse and induces action-aligned representations.<n>It learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.
  arXiv  Detail & Related papers  (2026-06-18T11:25:16Z)
• AstraNav-World: World Model for Foresight Control and Consistency [40.07910402326578]
  Embodied navigation in dynamic environments demands accurate foresight of how the world will evolve and how actions will unfold over time.<n>We propose AstraNav-World, an end-to-end world model that jointly reasons about future visual states and action sequences.<n>Our framework integrates a diffusion-based video generator with a vision-language policy, enabling synchronized rollouts.
  arXiv  Detail & Related papers  (2025-12-25T15:31:24Z)
• Flow and Depth Assisted Video Prediction with Latent Transformer [6.973908410173025]
  We present the first systematic study dedicated to occluded video prediction.<n>We use a standard multi-object latent transformer architecture to predict future frames, but modify this to incorporate information from depth and point-flow.<n>We find that when the prediction model is assisted with point flow and depth, it performs better in occluded scenarios and predicts more accurate background motion compared to models without the help of these modalities.
  arXiv  Detail & Related papers  (2025-11-20T15:54:33Z)
• Perpetua: Multi-Hypothesis Persistence Modeling for Semi-Static Environments [14.727014155729826]
  This paper introduces Perpetua, a method for modeling the dynamics of semi-static features.<n>We chain together mixtures of "persistence" and "emergence" filters to model the probability that features will disappear or reappear.<n>We find that Perpetua yields better accuracy than similar approaches while also being online adaptable and robust to missing observations.
  arXiv  Detail & Related papers  (2025-07-24T21:11:23Z)
• COME: Adding Scene-Centric Forecasting Control to Occupancy World Model [18.815436110557112]
  World models are critical for autonomous driving to simulate environmental dynamics and generate synthetic data.<n>Existing methods struggle to disentangle ego-vehicle motion (perspective shifts from scene evolvement)<n>We propose to separate environmental changes from ego-motion by leveraging the scene-centric coordinate systems.
  arXiv  Detail & Related papers  (2025-06-16T09:01:09Z)
• GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction [67.81475355852997]
  3D occupancy prediction is important for autonomous driving due to its comprehensive perception of the surroundings.<n>We propose a world-model-based framework to exploit the scene evolution for perception.<n>Our framework improves the performance of the single-frame counterpart by over 2% in mIoU without introducing additional computations.
  arXiv  Detail & Related papers  (2024-12-13T18:59:54Z)
• OPUS: Occupancy Prediction Using a Sparse Set [64.60854562502523]
  We present a framework to simultaneously predict occupied locations and classes using a set of learnable queries. OPUS incorporates a suite of non-trivial strategies to enhance model performance. Our lightest model achieves superior RayIoU on the Occ3D-nuScenes dataset at near 2x FPS, while our heaviest model surpasses previous best results by 6.1 RayIoU.
  arXiv  Detail & Related papers  (2024-09-14T07:44:22Z)
• Vectorized Representation Dreamer (VRD): Dreaming-Assisted Multi-Agent Motion-Forecasting [2.2020053359163305]
  We introduce VRD, a vectorized world model-inspired approach to the multi-agent motion forecasting problem. Our method combines a traditional open-loop training regime with a novel dreamed closed-loop training pipeline. Our model achieves state-of-the-art performance on the single prediction miss rate metric.
  arXiv  Detail & Related papers  (2024-06-20T15:34:17Z)
• GaussianPrediction: Dynamic 3D Gaussian Prediction for Motion Extrapolation and Free View Synthesis [71.24791230358065]
  We introduce a novel framework that empowers 3D Gaussian representations with dynamic scene modeling and future scenario synthesis. GaussianPrediction can forecast future states from any viewpoint, using video observations of dynamic scenes. Our framework shows outstanding performance on both synthetic and real-world datasets, demonstrating its efficacy in predicting and rendering future environments.
  arXiv  Detail & Related papers  (2024-05-30T06:47:55Z)
• Simple Video Generation using Neural ODEs [9.303957136142293]
  We learn latent variable models that predict the future in latent space and project back to pixels. We show that our approach yields promising results in the task of future frame prediction on the Moving MNIST dataset with 1 and 2 digits.
  arXiv  Detail & Related papers  (2021-09-07T19:03:33Z)
• FitVid: Overfitting in Pixel-Level Video Prediction [117.59339756506142]
  We introduce a new architecture, named FitVid, which is capable of severe overfitting on the common benchmarks. FitVid outperforms the current state-of-the-art models across four different video prediction benchmarks on four different metrics.
  arXiv  Detail & Related papers  (2021-06-24T17:20:21Z)
• VAE^2: Preventing Posterior Collapse of Variational Video Predictions in the Wild [131.58069944312248]
  We propose a novel VAE structure, dabbed VAE-in-VAE or VAE$2$. We treat part of the observed video sequence as a random transition state that bridges its past and future, and maximize the likelihood of a Markov Chain over the video sequence under all possible transition states. VAE$2$ can mitigate the posterior collapse problem to a large extent, as it breaks the direct dependence between future and observation and does not directly regress the determinate future provided by the training data.
  arXiv  Detail & Related papers  (2021-01-28T15:06:08Z)
• Future Urban Scenes Generation Through Vehicles Synthesis [90.1731992199415]
  We propose a deep learning pipeline to predict the visual future appearance of an urban scene. We follow a two stages approach, where interpretable information is included in the loop and each actor is modelled independently. We show the superiority of this approach over traditional end-to-end scene-generation methods on CityFlow.
  arXiv  Detail & Related papers  (2020-07-01T08:40:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.