Mining Multi-Modality Spatio-Temporal Cues for Video Important Person Identification

• URL: http://arxiv.org/abs/2605.28604v1
• Date: Wed, 27 May 2026 15:20:06 GMT
• Title: Mining Multi-Modality Spatio-Temporal Cues for Video Important Person Identification
• Abstract summary: Video Important Person (VIP) identification is aimed at automatically identifying the most influential individuals in videos.<n>We present Temporal-VIP, a large-scale rationale-annotated dataset consisting of 9,249 categories with aligned importance cues.
• Score: 66.15099321806618
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Identifying key individuals in video scenes is essential for applications such as automated video editing and intelligent surveillance. Current methods primarily focus on static images and immediate visual cues, overlooking the rich spatio-temporal information in videos. This leads to the phenomenon of Temporal Importance Shift (TIS), wherein individuals deemed significant in early frames may be demoted as the entire temporal context is considered. To address this, we introduce the Video Important Person (VIP) identification task, aimed at automatically identifying the most influential individuals in videos while providing textual rationales. We present Temporal-VIP, a large-scale rationale-annotated dataset consisting of 9,249 video segments across 11 categories with aligned importance rationales. To mitigate TIS, we develop the VIP-Net framework, which includes a Social Cue Encoder (SCE) for extracting multi-modal spatio-temporal cues, a Temporal Importance Rectifier (TIR) for hierarchical cue fusion and cross-modal alignment, and VIP Inference for ranking individuals. Experimental results show that VIP-Net achieves 67.3% accuracy, significantly outperforming state-of-the-art models (37.5%-53.9%) and yielding a mean rationale similarity of 0.63 to ground truth through feature-guided LLM refinement. The dataset and code are available at https://huggingface.co/datasets/yml2002/Temporal-VIP.

Related papers

• Object-Shot Enhanced Grounding Network for Egocentric Video [60.97916755629796]
  We propose OSGNet, an Object-Shot enhanced Grounding Network for egocentric video.<n>Specifically, we extract object information from videos to enrich video representation.<n>We analyze the frequent shot movements inherent to egocentric videos, leveraging these features to extract the wearer's attention information.
  arXiv  Detail & Related papers  (2025-05-07T09:20:12Z)
• MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understanding [0.8202721523031419]
  We propose a Multimodal hierarchical Attention focusing Self-reflective Reasoning framework for agent-based video understanding.<n>The key innovation lies in its ability to detect and prioritize segments of videos that are highly relevant to the query.
  arXiv  Detail & Related papers  (2025-04-24T02:54:40Z)
• Video-based Person Re-identification with Long Short-Term Representation Learning [101.62570747820541]
  Video-based person Re-Identification (V-ReID) aims to retrieve specific persons from raw videos captured by non-overlapped cameras. We propose a novel deep learning framework named Long Short-Term Representation Learning (LSTRL) for effective V-ReID.
  arXiv  Detail & Related papers  (2023-08-07T16:22:47Z)
• Deeply-Coupled Convolution-Transformer with Spatial-temporal Complementary Learning for Video-based Person Re-identification [91.56939957189505]
  We propose a novel spatial-temporal complementary learning framework named Deeply-Coupled Convolution-Transformer (DCCT) for high-performance video-based person Re-ID. Our framework could attain better performances than most state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-04-27T12:16:44Z)
• Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition [56.91538445510214]
  We present a learning algorithm for human activity recognition in videos. Our approach is designed for UAV videos, which are mainly acquired from obliquely placed dynamic cameras. We conduct extensive experiments on the UAV Human dataset and the NEC Drone dataset.
  arXiv  Detail & Related papers  (2022-09-15T22:16:52Z)
• Stacked Temporal Attention: Improving First-person Action Recognition by Emphasizing Discriminative Clips [39.29955809641396]
  Many backgrounds or noisy frames in a first-person video can distract an action recognition model during its learning process. Previous works explored to address this problem by applying temporal attention but failed to consider the global context of the full video. We propose a simple yet effective Stacked Temporal Attention Module (STAM) to compute temporal attention based on the global knowledge across clips.
  arXiv  Detail & Related papers  (2021-12-02T08:02:35Z)
• Dense Interaction Learning for Video-based Person Re-identification [75.03200492219003]
  We propose a hybrid framework, Dense Interaction Learning (DenseIL), to tackle video-based person re-ID difficulties. DenseIL contains a CNN encoder and a Dense Interaction (DI) decoder. Our experiments consistently and significantly outperform all the state-of-the-art methods on multiple standard video-based re-ID datasets.
  arXiv  Detail & Related papers  (2021-03-16T12:22:08Z)
• Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition [14.07119502083967]
  Driver's activities are different since they are executed by the same subject with similar body parts movements, resulting in subtle changes. Our model is named Coarse Temporal Attention Network (CTA-Net), in which coarse temporal branches are introduced in a trainable glimpse. The model then uses an innovative attention mechanism to generate high-level action specific contextual information for activity recognition.
  arXiv  Detail & Related papers  (2021-01-17T10:15:37Z)
• Personalizing Fast-Forward Videos Based on Visual and Textual Features from Social Network [9.353403626477135]
  We present a new approach to automatically creating personalized fast-forward videos for First-Person Videos (FPVs) Our approach explores the availability of text-centric data from the user's social networks to infer her/his topics of interest and assigns scores to the input frames according to her/his preferences.
  arXiv  Detail & Related papers  (2019-12-29T14:09:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.