Video-CoE: Reinforcing Video Event Prediction via Chain of Events

• URL: http://arxiv.org/abs/2603.14935v1
• Date: Mon, 16 Mar 2026 07:38:15 GMT
• Title: Video-CoE: Reinforcing Video Event Prediction via Chain of Events
• Abstract summary: VEP requires the model to perform fine-grained temporal modeling of videos and establish logical relationships between videos and future events.<n>textbfChain textbfof textbfEvents (textbfCoE) paradigm, which constructs temporal event chains to implicitly enforce MLLM.<n>Codes and models will be released soon.
• Score: 20.58429395226725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite advances in the application of MLLMs for various video tasks, video event prediction (VEP) remains relatively underexplored. VEP requires the model to perform fine-grained temporal modeling of videos and establish logical relationships between videos and future events, which current MLLMs still struggle with. In this work, we first present a comprehensive evaluation of current leading MLLMs on the VEP task, revealing the reasons behind their inaccurate predictions, including lack of logical reasoning ability for future events prediction and insufficient utilization of visual information. To address these challenges, we propose \textbf{C}hain \textbf{o}f \textbf{E}vents (\textbf{CoE}) paradigm, which constructs temporal event chains to implicitly enforce MLLM focusing on the visual content and the logical connections between videos and future events, incentivizing model's reasoning capability with multiple training protocols. Experimental results on public benchmarks demonstrate that our method outperforms both leading open-source and commercial MLLMs, establishing a new state-of-the-art on the VEP task. Codes and models will be released soon.

Related papers

• Benchmarking Visual State Tracking in Multimodal Video Understanding [67.73383974166462]
  We introduce Visual STAte Tracking benchmark (VSTAT) to diagnose visual state tracking in Multimodal Large Language Models (MLLMs)<n>VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment.<n>We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track.
  arXiv  Detail & Related papers  (2026-06-02T17:12:05Z)
• Towards Effective Long-Video Event Prediction via Multi-Level Event Semantics Mining [10.37877709276133]
  We propose VISTA, a multi-level event semantics mining framework for long-video event prediction.<n>Initially, VISTA applies a character-centric visual prompt to precisely extract event-related visual details, enhancing detail-level semantics.<n>It then employs a knowledge-enhanced iterative retrieval strategy, guiding the LLM to progressively construct logically coherent event chains.<n>Ultimately, VISTA adopts a human-like propose-then-retrieve strategy to generate diverse future-oriented proposals and integrate multi-level clues, producing robust and accurate predictions.
  arXiv  Detail & Related papers  (2026-05-29T09:38:43Z)
• LEAF: A Living Benchmark for Event-Augmented Forecasting [78.94019460534442]
  LEAF is the first living benchmark for event-augmented forecasting tasks.<n>We evaluate state-of-the-art proprietary and open-weight LLMs.<n>We find that LLMs achieve better performance on equities they confidently identify as more predictable.
  arXiv  Detail & Related papers  (2026-05-09T03:17:59Z)
• Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO [20.96275248557104]
  Video-Next-Event Prediction (VNEP) requires dynamic video responses to predict the next event in text.<n>We introduce VANS, a model that leverages reinforcement learning to align a Vision-Language Model (VLM) with a Video Diffusion Model (VDM) for VNEP.<n>The core of VANS is our proposed Joint-GRPO that orchestrates the VLM and VDM to function as a unit.<n> Experiments on procedural and predictive benchmarks demonstrate that VANS achieves state-of-the-art performance in both video event prediction and visualization.
  arXiv  Detail & Related papers  (2025-11-20T18:59:44Z)
• EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction [7.250942234168963]
  We introduce AVEP (Action-centric Video Event Prediction), a task that distinguishes itself from existing video prediction tasks.<n>We present a large structured dataset, which consists of about $35K$ annotated videos and more than $178K$ video clips of event.<n>We propose EventFormer, a node-graph hierarchical attention based video event prediction model, which can capture both the relationships between events and their arguments.
  arXiv  Detail & Related papers  (2025-10-19T04:46:45Z)
• Aligning Effective Tokens with Video Anomaly in Large Language Models [42.99603812716817]
  We propose VA-GPT, a novel MLLM designed for summarizing and localizing abnormal events in various videos.<n>Our approach efficiently aligns effective tokens between visual encoders and LLMs through two key proposed modules.<n>We construct an instruction-following dataset specifically for fine-tuning video-anomaly-aware MLLMs.
  arXiv  Detail & Related papers  (2025-08-08T14:30:05Z)
• Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Grounding [49.51013055630857]
  We tackle the task of online video temporal grounding (OnVTG), which requires the model to locate events related to a given text query within a video stream.<n>Unlike regular video temporal grounding, OnVTG requires the model to make predictions without observing future frames.<n>We propose an event-based OnVTG framework that makes predictions based on event proposals that model event-level information with various durations.
  arXiv  Detail & Related papers  (2025-08-06T15:33:49Z)
• TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understanding [25.675553077419274]
  Multimodal Language Models (MLLMs) have demonstrated significant progress in vision tasks, yet they still face challenges when processing long-duration inputs.<n>We propose Temporal Sampling Policy Optimization (TSPO), advancing MLLMs' long-form video-language understanding via reinforcement learning.<n>Our TSPO state-of-the-art performance across multiple long video understanding benchmarks, and shows transferable ability across different cutting-edge Video-MLLMs.
  arXiv  Detail & Related papers  (2025-08-06T12:03:36Z)
• Fostering Video Reasoning via Next-Event Prediction [61.70045315542766]
  We propose next-event prediction (NEP) as a learning task that harnesses future video segments as a rich, self-supervised signal to foster temporal reasoning.<n>To support this task, we curate V1-33K, a dataset comprising 33,000 automatically extracted video segments spanning diverse real-world scenarios.<n> Experiments validate that NEP offers a scalable and effective training paradigm for fostering temporal reasoning in MLLMs.
  arXiv  Detail & Related papers  (2025-05-28T15:13:34Z)
• Grounding Partially-Defined Events in Multimodal Data [61.0063273919745]
  We introduce a multimodal formulation for partially-defined events and cast the extraction of these events as a three-stage span retrieval task. We propose a benchmark for this task, MultiVENT-G, that consists of 14.5 hours of densely annotated current event videos and 1,168 text documents, containing 22.8K labeled event-centric entities. Results illustrate the challenges that abstract event understanding poses and demonstrates promise in event-centric video-language systems.
  arXiv  Detail & Related papers  (2024-10-07T17:59:48Z)
• EventHallusion: Diagnosing Event Hallucinations in Video LLMs [42.66453293963568]
  Multimodal Large Language Models (MLLMs) have made significant progress in the video comprehension field.<n>We propose EventHallusion, a novel benchmark that focuses on assessing the VideoLLMs' hallucination toward event.<n>We also propose a simple yet effective method, called Temporal Contrastive Decoding (TCD), to tackle the hallucination problems of VideoLLMs.
  arXiv  Detail & Related papers  (2024-09-25T03:49:46Z)
• Towards Event-oriented Long Video Understanding [101.48089908037888]
  Event-Bench is an event-oriented long video understanding benchmark built on existing datasets and human annotations. VIM is a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions.
  arXiv  Detail & Related papers  (2024-06-20T09:14:19Z)
• Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models [20.33971942003996]
  This study introduces an innovative method to address event-level hallucinations in MLLMs. We propose a unique mechanism that decomposes on-demand event queries into iconic actions. We employ models like CLIP and BLIP2 to predict specific timestamps for event occurrences.
  arXiv  Detail & Related papers  (2024-01-18T10:18:48Z)
• PALM: Predicting Actions through Language Models [74.10147822693791]
  We introduce PALM, an approach that tackles the task of long-term action anticipation. Our method incorporates an action recognition model to track previous action sequences and a vision-language model to articulate relevant environmental details. Our experimental results demonstrate that PALM surpasses the state-of-the-art methods in the task of long-term action anticipation.
  arXiv  Detail & Related papers  (2023-11-29T02:17:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.