FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

• URL: http://arxiv.org/abs/2601.13836v1
• Date: Tue, 20 Jan 2026 10:47:20 GMT
• Title: FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs
• Abstract summary: We introduce Future Omni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments.<n>Future Omni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs.<n> Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios.
• Score: 94.33687833634275
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although Multimodal Large Language Models (MLLMs) demonstrate strong omni-modal perception, their ability to forecast future events from audio-visual cues remains largely unexplored, as existing benchmarks focus mainly on retrospective understanding. To bridge this gap, we introduce FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments. The evaluated models are required to perform cross-modal causal and temporal reasoning, as well as effectively leverage internal knowledge to predict future events. FutureOmni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs across 8 primary domains. Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios, with the best accuracy of 64.8% achieved by Gemini 3 Flash. To mitigate this limitation, we curate a 7K-sample instruction-tuning dataset and propose an Omni-Modal Future Forecasting (OFF) training strategy. Evaluations on FutureOmni and popular audio-visual and video-only benchmarks demonstrate that OFF enhances future forecasting and generalization. We publicly release all code (https://github.com/OpenMOSS/FutureOmni) and datasets (https://huggingface.co/datasets/OpenMOSS-Team/FutureOmni).

Related papers

• Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction [43.50136583739717]
  We introduce Future-L1, an interleaved latent visual reasoning framework.<n>It lets an MLLM alternate between language tokens and continuous latent visual spans during autoregressive decoding.<n>Future-L1 achieves new state-of-the-art results on both benchmarks.
  arXiv  Detail & Related papers  (2026-06-04T06:53:19Z)
• LLM-Guided Future Hypotheses for Horizon-Aware Exploration in Multi-Step Robot Manipulation [5.637033593506126]
  Multi-step robot manipulation requires acting under uncertainty about how the scene will evolve.<n>We study whether short-horizon, task-consistent future videos can provide useful structured priors for control and reinforcement-learning fine-tuning.
  arXiv  Detail & Related papers  (2026-05-28T12:49:37Z)
• OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding [17.161087166882744]
  We present OmniPro, the first benchmark to jointly evaluate omni-modal perception, proactive responding, and diverse video understanding tasks.<n>It comprises 2,700 human-verified samples spanning 9 sub-tasks and 3 cognitive levels, covering 6 basic video understanding capabilities.<n> Notably, 84% of samples require audio signals (speech or non-speech), and each sample is annotated with modality-isolation labels to enable fine-grained multimodal analysis.
  arXiv  Detail & Related papers  (2026-05-18T15:55:22Z)
• Future Optical Flow Prediction Improves Robot Control & Video Generation [100.87884718953099]
  We introduce FOFPred, a novel optical flow forecasting model featuring a unified Vision-Language Model (VLM) and Diffusion architecture.<n>Our model is trained on web-scale human activity data-a highly scalable but unstructured source.<n> Evaluations across robotic manipulation and video generation under language-driven settings establish the cross-domain versatility of FOFPred.
  arXiv  Detail & Related papers  (2026-01-15T18:49:48Z)
• OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM [146.029449832893]
  We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM.<n>For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings.
  arXiv  Detail & Related papers  (2025-10-17T17:59:59Z)
• PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling [95.2927277964409]
  We introduce Online Audio-Visual Event Parsing (On-AVEP), a novel paradigm for parsing audio, visual, and audio-visual events by sequentially analyzing incoming video streams.<n>We propose the Predictive Future Modeling framework featured by (a) predictive multimodal future modeling to infer and integrate beneficial future audio-visual cues.<n>Experiments show PreFM significantly outperforms state-of-the-art methods by a large margin with significantly fewer parameters.
  arXiv  Detail & Related papers  (2025-05-29T06:46:19Z)
• Fostering Video Reasoning via Next-Event Prediction [61.70045315542766]
  We propose next-event prediction (NEP) as a learning task that harnesses future video segments as a rich, self-supervised signal to foster temporal reasoning.<n>To support this task, we curate V1-33K, a dataset comprising 33,000 automatically extracted video segments spanning diverse real-world scenarios.<n> Experiments validate that NEP offers a scalable and effective training paradigm for fostering temporal reasoning in MLLMs.
  arXiv  Detail & Related papers  (2025-05-28T15:13:34Z)
• Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals [4.550158880305567]
  We propose an automatic data collection pipeline that allows us to collect and annotate over 210 hours of human conversation videos.<n>We construct a Multi-Modal Face-to-Face dataset, including over 1.5M words and corresponding turn-taking and backchannel annotations from approximately 20M frames.<n>We present an end-to-end framework that predicts the probability of turn-taking and backchannel actions from multi-modal signals.
  arXiv  Detail & Related papers  (2025-05-19T03:08:30Z)
• Ola: Pushing the Frontiers of Omni-Modal Language Model [88.72389428177942]
  We present Ola, an omni-modal language model that achieves competitive performance across image, video, and audio understanding.<n>Ola incorporates advanced visual understanding and audio recognition capabilities through several critical and effective improvements.<n>We aim to make Ola a fully open omni-modal understanding solution to advance future research in this emerging field.
  arXiv  Detail & Related papers  (2025-02-06T18:59:55Z)
• Video Prediction at Multiple Scales with Hierarchical Recurrent Networks [24.536256844130996]
  We propose a novel video prediction model able to forecast future possible outcomes of different levels of granularity simultaneously. By combining spatial and temporal downsampling, MSPred is able to efficiently predict abstract representations over long time horizons. In our experiments, we demonstrate that our proposed model accurately predicts future video frames as well as other representations on various scenarios.
  arXiv  Detail & Related papers  (2022-03-17T13:08:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.