Towards Long-Form Spatio-Temporal Video Grounding

• URL: http://arxiv.org/abs/2602.23294v1
• Date: Thu, 26 Feb 2026 18:04:09 GMT
• Title: Towards Long-Form Spatio-Temporal Video Grounding
• Authors: Xin Gu, Bing Fan, Jiali Yao, Zhipeng Zhang, Yan Huang, Cheng Han, Heng Fan, Libo Zhang,
• Abstract summary: Long-term videos contain much longer temporal span and spans more irrelevant information.<n>Unlike conventional STVG methods that require the entire video sequence to make predictions at once, ART-STVG treats the video as streaming input frames and processes sequentially.<n>Since memories from different moments are not always relevant to the current frame, we introduce simple yet effective memory selection strategies.
• Score: 37.582576273199955
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In real scenarios, videos can span several minutes or even hours. However, existing research on spatio-temporal video grounding (STVG), given a textual query, mainly focuses on localizing targets in short videos of tens of seconds, typically less than one minute, which limits real-world applications. In this paper, we explore Long-Form STVG (LF-STVG), which aims to locate targets in long-term videos. Compared with short videos, long-term videos contain much longer temporal spans and more irrelevant information, making it difficult for existing STVG methods that process all frames at once. To address this challenge, we propose an AutoRegressive Transformer architecture for LF-STVG, termed ART-STVG. Unlike conventional STVG methods that require the entire video sequence to make predictions at once, ART-STVG treats the video as streaming input and processes frames sequentially, enabling efficient handling of long videos. To model spatio-temporal context, we design spatial and temporal memory banks and apply them to the decoders. Since memories from different moments are not always relevant to the current frame, we introduce simple yet effective memory selection strategies to provide more relevant information to the decoders, significantly improving performance. Furthermore, instead of parallel spatial and temporal localization, we propose a cascaded spatio-temporal design that connects the spatial decoder to the temporal decoder, allowing fine-grained spatial cues to assist complex temporal localization in long videos. Experiments on newly extended LF-STVG datasets show that ART-STVG significantly outperforms state-of-the-art methods, while achieving competitive performance on conventional short-form STVG.

Related papers

• TV-RAG: A Temporal-aware and Semantic Entropy-Weighted Framework for Long Video Retrieval and Understanding [14.570869250170139]
  TV-RAG is a training-free architecture that couples temporal alignment with entropy-guided semantics to improve long-video reasoning.<n>By weaving these temporal and semantic signals together, TV-RAG realises a dual-level reasoning routine that can be grafted onto any LVLM without re-training or fine-tuning.
  arXiv  Detail & Related papers  (2025-12-29T14:10:22Z)
• Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning [41.30900315121155]
  multimodal large language models (LMs) underperform on STVG due to misaligned training objectives and weak fine-grained fine-word alignment in standard visual encoders.<n>We propose STVG-o1, the first framework that enables off-the-shelf MLLMs to achieve state-of-the-temporal STVG performance without architectural modifications.
  arXiv  Detail & Related papers  (2025-11-26T13:21:15Z)
• Exploiting Temporal State Space Sharing for Video Semantic Segmentation [53.8810901249897]
  Video semantic segmentation (VSS) plays a vital role in understanding the temporal evolution of scenes.<n>Traditional methods often segment videos frame-by-frame or in a short temporal window, leading to limited temporal context, redundant computations, and heavy memory requirements.<n>We introduce a Temporal Video State Space Sharing architecture to leverage Mamba state space models for temporal feature sharing.<n>Our model features a selective gating mechanism that efficiently propagates relevant information across video frames, eliminating the need for a memory-heavy feature pool.
  arXiv  Detail & Related papers  (2025-03-26T01:47:42Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• Video LLMs for Temporal Reasoning in Long Videos [7.2900856926028155]
  TemporalVLM is a video large language model capable of effective temporal reasoning and fine-grained understanding in long videos.<n>Our approach includes a visual encoder for mapping a long-term input video into features which are time-aware and contain both local and global cues.<n>To facilitate the evaluation of TemporalVLM, we present a large-scale long video dataset of industry assembly processes.
  arXiv  Detail & Related papers  (2024-12-04T00:50:33Z)
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding [65.46303012350207]
  LongVU is an adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. We leverage DINOv2 features to remove redundant frames that exhibit high similarity. We perform spatial token reduction across frames based on their temporal dependencies.
  arXiv  Detail & Related papers  (2024-10-22T21:21:37Z)
• FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance [3.6519202494141125]
  We introduce FancyVideo, an innovative video generator that improves the existing text-control mechanism.<n>CTGM incorporates the Temporal Information (TII) and Temporal Affinity Refiner (TAR) at the beginning and end of cross-attention.<n>Our approach achieves state-of-the-art T2V generation results on the EvalCrafter benchmark.
  arXiv  Detail & Related papers  (2024-08-15T14:47:44Z)
• Efficient Long-Short Temporal Attention Network for Unsupervised Video Object Segmentation [23.645412918420906]
  Unsupervised Video Object (VOS) aims at identifying the contours of primary foreground objects in videos without any prior knowledge. Previous methods do not fully use spatial-temporal context and fail to tackle this challenging task in real-time. This motivates us to develop an efficient Long-Short Temporal Attention network (termed LSTA) for unsupervised VOS task from a holistic view.
  arXiv  Detail & Related papers  (2023-09-21T01:09:46Z)
• TubeDETR: Spatio-Temporal Video Grounding with Transformers [89.71617065426146]
  We consider the problem of encoder localizing a-temporal tube in a video corresponding to a given text query. To address this task, we propose TubeDETR, a transformer- architecture inspired by the recent success of such models for text-conditioned object detection.
  arXiv  Detail & Related papers  (2022-03-30T16:31:49Z)
• Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences [107.0776836117313]
  Given an un-trimmed video and a declarative/interrogative sentence, STVG aims to localize the-temporal tube of the object queried. Existing methods cannot tackle the STVG task due to the ineffective tube pre-generation and the lack of novel object relationship modeling. We present a declarative-Temporal Graph Reasoning Network (STGRN) for this task.
  arXiv  Detail & Related papers  (2020-01-19T19:53:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.