Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining

• URL: http://arxiv.org/abs/2603.22953v1
• Date: Tue, 24 Mar 2026 08:48:15 GMT
• Title: Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining
• Abstract summary: ClusterSTM is a Cluster-Wise Spatio-Temporal Masking strategy for efficient video-language pretraining.<n>We introduce a video-text relevance reconstruction objective that aligns high-level multimodal semantics beyond conventional visual reconstruction.
• Score: 86.49790441700195
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale video-language pretraining enables strong generalization across multimodal tasks but often incurs prohibitive computational costs. Although recent advances in masked visual modeling help mitigate this issue, they still suffer from two fundamental limitations: severe visual information loss under high masking ratios and temporal information leakage caused by inter-frame correlations. To address these challenges, we propose ClusterSTM, a Cluster-Wise Spatio-Temporal Masking strategy for efficient video-language pretraining. ClusterSTM first performs intra-frame clustering to partition visual tokens into multiple semantically independent clusters, then conducts cluster-wise masking by retaining the token with the highest temporal density within each cluster. Our masking strategy ensure that the retained tokens capture holistic video content while exhibit strong temporal correlation. Additionally, we introduce a video-text relevance reconstruction objective that aligns high-level multimodal semantics beyond conventional visual reconstruction. Extensive experiments across multiple benchmarks demonstrate that ClusterSTM achieves superior performance on video-text retrieval, video question answering, and video captioning tasks, establishing a new state-of-the-art among efficient video-language models.

Related papers

• SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM [36.28285195488772]
  Large language models (LLMs) have demonstrated exceptional capabilities in text understanding.<n>Vid-LLMs struggle to simultaneously retain high-quality frame-level semantic information.<n>This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding.
  arXiv  Detail & Related papers  (2026-02-03T14:39:16Z)
• MultiHateLoc: Towards Temporal Localisation of Multimodal Hate Content in Online Videos [22.175314789730667]
  MultiHateLoc is a framework for weakly-supervised multimodal hate localisation.<n>It produces fine-grained, interpretable frame-level predictions.<n> Experiments on HateMM and MultiHateClip show that our method achieves state-of-the-art performance in the localisation task.
  arXiv  Detail & Related papers  (2025-12-11T08:18:22Z)
• Xiaoice: Training-Free Video Understanding via Self-Supervised Spatio-Temporal Clustering of Semantic Features [10.21556794551883]
  This paper introduces a novel, training-free framework for video understanding that circumvents end-to-end training.<n>Our core idea is to video understanding as a self-supervised-temporal clustering within a high-dimensional feature space.<n>This approach provides an effective, interpretable, and model-agnostic pathway for zero-shot, automated structural analysis of video content.
  arXiv  Detail & Related papers  (2025-10-19T10:13:34Z)
• Efficient Multi-modal Large Language Models via Visual Token Grouping [55.482198808206284]
  High-resolution images and videos pose a barrier to their broader adoption.<n> compressing vision tokens in MLLMs has emerged as a promising approach to reduce inference costs.<n>We introduce VisToG, a novel grouping mechanism that leverages the capabilities of pre-trained vision encoders to group similar image segments.
  arXiv  Detail & Related papers  (2024-11-26T09:36:02Z)
• Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation [62.275143240798236]
  Video semantic segmentation dataset has limited categories per video. Less than 10% of queries could be matched to receive meaningful gradient updates during VSS training. Our method achieves state-of-the-art performance on the latest challenging VSS benchmark VSPW without bells and whistles.
  arXiv  Detail & Related papers  (2023-09-14T20:31:06Z)
• CenterCLIP: Token Clustering for Efficient Text-Video Retrieval [67.21528544724546]
  In CLIP, the essential visual tokenization process, which produces discrete visual token sequences, generates many homogeneous tokens due to the redundancy nature of consecutive frames in videos. This significantly increases computation costs and hinders the deployment of video retrieval models in web applications. In this paper, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones.
  arXiv  Detail & Related papers  (2022-05-02T12:02:09Z)
• Face, Body, Voice: Video Person-Clustering with Multiple Modalities [85.0282742801264]
  Previous methods focus on the narrower task of face-clustering. Most current datasets evaluate only the task of face-clustering, rather than person-clustering. We introduce a Video Person-Clustering dataset, for evaluating multi-modal person-clustering.
  arXiv  Detail & Related papers  (2021-05-20T17:59:40Z)
• Generating Masks from Boxes by Mining Spatio-Temporal Consistencies in Videos [159.02703673838639]
  We introduce a method for generating segmentation masks from per-frame bounding box annotations in videos. We use our resulting accurate masks for weakly supervised training of video object segmentation (VOS) networks. The additional data provides substantially better generalization performance leading to state-of-the-art results in both the VOS and more challenging tracking domain.
  arXiv  Detail & Related papers  (2021-01-06T18:56:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.