Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction

• URL: http://arxiv.org/abs/2606.05769v1
• Date: Thu, 04 Jun 2026 06:53:19 GMT
• Title: Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction
• Abstract summary: We introduce Future-L1, an interleaved latent visual reasoning framework.<n>It lets an MLLM alternate between language tokens and continuous latent visual spans during autoregressive decoding.<n>Future-L1 achieves new state-of-the-art results on both benchmarks.
• Score: 43.50136583739717
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video event prediction (VEP) requires models to infer unobserved future states from partial video evidence. Existing video MLLMs usually verbalize intermediate future reasoning in text space: once visual evidence is verbalized, fine-grained motion, geometry, and interaction cues can be lost, leading to plausible but visually ungrounded hallucinations. We introduce Future-L1, an interleaved latent visual reasoning framework that lets an MLLM alternate between language tokens and continuous latent visual spans during autoregressive decoding. To train this capability, we construct Future-L1-50K by selecting examples where future visual hints help prediction and align latent states to future-frame embeddings, then further optimize sampled latent trajectories with LA-DAPO, a latent-aware RL objective with outcome-contrastive and temporal-diversity rewards. Future-L1 achieves new state-of-the-art results on both benchmarks: on FutureBench, it improves Qwen3-VL-8B from 61.0 to 85.4 and exceeds the previous best Video-CoE by 10.4 points; on TwiFF-Bench, it improves the average score from 2.44 to 3.04. These results suggest that future-oriented video reasoning benefits from preserving intermediate visual semantics in latent space rather than translating every reasoning step into text.

Related papers

• Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning [15.521458272682308]
  Multimodal large language models increasingly use visual chain-of-thought (Visual CoT) to reason about spatial, temporal, and embodied environments.<n>We ask whether models can learn to think visually during training while reasoning directly at inference.<n>We introduce Internalized Visual Thinking (IVT), a framework that jointly optimize textual prediction and next-embedding prediction over unlabeled videos.
  arXiv  Detail & Related papers  (2026-08-16T17:35:38Z)
• Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes [39.17422574355306]
  We introduce Predictive State Retrieval (PSR), a task in which a model observes a short video prefix and a temporal question about an object's future state.<n>PSR combines anticipation with cross-instance retrieval across multiple temporal horizons.<n>We also propose LFTR, a lightweight retriever with frozen encoders that predicts a question- and horizon-conditioned future latent.
  arXiv  Detail & Related papers  (2026-08-05T04:12:00Z)
• From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models [67.00247936789127]
  Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions.<n>We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs.
  arXiv  Detail & Related papers  (2026-06-13T18:00:42Z)
• FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs [94.33687833634275]
  We introduce Future Omni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments.<n>Future Omni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs.<n> Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios.
  arXiv  Detail & Related papers  (2026-01-20T10:47:20Z)
• Fostering Video Reasoning via Next-Event Prediction [61.70045315542766]
  We propose next-event prediction (NEP) as a learning task that harnesses future video segments as a rich, self-supervised signal to foster temporal reasoning.<n>To support this task, we curate V1-33K, a dataset comprising 33,000 automatically extracted video segments spanning diverse real-world scenarios.<n> Experiments validate that NEP offers a scalable and effective training paradigm for fostering temporal reasoning in MLLMs.
  arXiv  Detail & Related papers  (2025-05-28T15:13:34Z)
• Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023 [100.32802766127776]
  Palm is a solution to the Long-Term Action Anticipation task utilizing vision-language and large language models. It predicts future actions based on frame descriptions and action labels extracted from the input videos.
  arXiv  Detail & Related papers  (2023-06-28T20:33:52Z)
• Video Prediction at Multiple Scales with Hierarchical Recurrent Networks [24.536256844130996]
  We propose a novel video prediction model able to forecast future possible outcomes of different levels of granularity simultaneously. By combining spatial and temporal downsampling, MSPred is able to efficiently predict abstract representations over long time horizons. In our experiments, we demonstrate that our proposed model accurately predicts future video frames as well as other representations on various scenarios.
  arXiv  Detail & Related papers  (2022-03-17T13:08:28Z)
• VAE^2: Preventing Posterior Collapse of Variational Video Predictions in the Wild [131.58069944312248]
  We propose a novel VAE structure, dabbed VAE-in-VAE or VAE$2$. We treat part of the observed video sequence as a random transition state that bridges its past and future, and maximize the likelihood of a Markov Chain over the video sequence under all possible transition states. VAE$2$ can mitigate the posterior collapse problem to a large extent, as it breaks the direct dependence between future and observation and does not directly regress the determinate future provided by the training data.
  arXiv  Detail & Related papers  (2021-01-28T15:06:08Z)
• Learning to Anticipate Egocentric Actions by Imagination [60.21323541219304]
  We study the egocentric action anticipation task, which predicts future action seconds before it is performed for egocentric videos. Our method significantly outperforms previous methods on both the seen test set and the unseen test set of the EPIC Kitchens Action Anticipation Challenge.
  arXiv  Detail & Related papers  (2021-01-13T08:04:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.