EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction

• URL: http://arxiv.org/abs/2510.21786v1
• Date: Sun, 19 Oct 2025 04:46:45 GMT
• Title: EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction
• Authors: Qile Su, Shoutai Zhu, Shuai Zhang, Baoyu Liang, Chao Tong,
• Abstract summary: We introduce AVEP (Action-centric Video Event Prediction), a task that distinguishes itself from existing video prediction tasks.<n>We present a large structured dataset, which consists of about $35K$ annotated videos and more than $178K$ video clips of event.<n>We propose EventFormer, a node-graph hierarchical attention based video event prediction model, which can capture both the relationships between events and their arguments.
• Score: 7.250942234168963
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Script event induction, which aims to predict the subsequent event based on the context, is a challenging task in NLP, achieving remarkable success in practical applications. However, human events are mostly recorded and presented in the form of videos rather than scripts, yet there is a lack of related research in the realm of vision. To address this problem, we introduce AVEP (Action-centric Video Event Prediction), a task that distinguishes itself from existing video prediction tasks through its incorporation of more complex logic and richer semantic information. We present a large structured dataset, which consists of about $35K$ annotated videos and more than $178K$ video clips of event, built upon existing video event datasets to support this task. The dataset offers more fine-grained annotations, where the atomic unit is represented as a multimodal event argument node, providing better structured representations of video events. Due to the complexity of event structures, traditional visual models that take patches or frames as input are not well-suited for AVEP. We propose EventFormer, a node-graph hierarchical attention based video event prediction model, which can capture both the relationships between events and their arguments and the coreferencial relationships between arguments. We conducted experiments using several SOTA video prediction models as well as LVLMs on AVEP, demonstrating both the complexity of the task and the value of the dataset. Our approach outperforms all these video prediction models. We will release the dataset and code for replicating the experiments and annotations.

Related papers

• Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Grounding [49.51013055630857]
  We tackle the task of online video temporal grounding (OnVTG), which requires the model to locate events related to a given text query within a video stream.<n>Unlike regular video temporal grounding, OnVTG requires the model to make predictions without observing future frames.<n>We propose an event-based OnVTG framework that makes predictions based on event proposals that model event-level information with various durations.
  arXiv  Detail & Related papers  (2025-08-06T15:33:49Z)
• VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in Videos [6.442765801124304]
  We propose the task of video event understanding that extracts event scripts and makes predictions with these scripts from videos.<n>To support this task, we introduce VidEvent, a large-scale dataset containing over 23,000 well-labeled events.<n>The dataset was created through a meticulous annotation process, ensuring high-quality and reliable event data.
  arXiv  Detail & Related papers  (2025-06-03T05:12:48Z)
• Grounding Partially-Defined Events in Multimodal Data [61.0063273919745]
  We introduce a multimodal formulation for partially-defined events and cast the extraction of these events as a three-stage span retrieval task. We propose a benchmark for this task, MultiVENT-G, that consists of 14.5 hours of densely annotated current event videos and 1,168 text documents, containing 22.8K labeled event-centric entities. Results illustrate the challenges that abstract event understanding poses and demonstrates promise in event-centric video-language systems.
  arXiv  Detail & Related papers  (2024-10-07T17:59:48Z)
• EA-VTR: Event-Aware Video-Text Retrieval [97.30850809266725]
  Event-Aware Video-Text Retrieval model achieves powerful video-text retrieval ability through superior video event awareness. EA-VTR can efficiently encode frame-level and video-level visual representations simultaneously, enabling detailed event content and complex event temporal cross-modal alignment.
  arXiv  Detail & Related papers  (2024-07-10T09:09:58Z)
• Event-aware Video Corpus Moment Retrieval [79.48249428428802]
  Video Corpus Moment Retrieval (VCMR) is a practical video retrieval task focused on identifying a specific moment within a vast corpus of untrimmed videos. Existing methods for VCMR typically rely on frame-aware video retrieval, calculating similarities between the query and video frames to rank videos. We propose EventFormer, a model that explicitly utilizes events within videos as fundamental units for video retrieval.
  arXiv  Detail & Related papers  (2024-02-21T06:55:20Z)
• SPOT! Revisiting Video-Language Models for Event Understanding [31.49859545456809]
  We introduce SPOT Prober, to benchmark existing video-language models's capacities of distinguishing event-level discrepancies. We evaluate the existing video-language models with these positive and negative captions and find they fail to distinguish most of the manipulated events. Based on our findings, we propose to plug in these manipulated event captions as hard negative samples and find them effective in enhancing models for event understanding.
  arXiv  Detail & Related papers  (2023-11-21T18:43:07Z)
• CLIP-Event: Connecting Text and Images with Event Structures [123.31452120399827]
  We propose a contrastive learning framework to enforce vision-language pretraining models. We take advantage of text information extraction technologies to obtain event structural knowledge. Experiments show that our zero-shot CLIP-Event outperforms the state-of-the-art supervised model in argument extraction.
  arXiv  Detail & Related papers  (2022-01-13T17:03:57Z)
• Team RUC_AIM3 Technical Report at Activitynet 2020 Task 2: Exploring Sequential Events Detection for Dense Video Captioning [63.91369308085091]
  We propose a novel and simple model for event sequence generation and explore temporal relationships of the event sequence in the video. The proposed model omits inefficient two-stage proposal generation and directly generates event boundaries conditioned on bi-directional temporal dependency in one pass. The overall system achieves state-of-the-art performance on the dense-captioning events in video task with 9.894 METEOR score on the challenge testing set.
  arXiv  Detail & Related papers  (2020-06-14T13:21:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.