Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding

• URL: http://arxiv.org/abs/2604.08014v2
• Date: Tue, 14 Apr 2026 01:55:33 GMT
• Title: Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding
• Abstract summary: textbfBridge-STG is an end-to-end framework that decouples temporal and spatial localization while maintaining semantic coherence.<n>Bridge-STG achieves state-of-the-art performance among MLLM-based methods.
• Score: 6.673833357112721
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Spatio-Temporal Video Grounding requires jointly localizing target objects across both temporal and spatial dimensions based on natural language queries, posing fundamental challenges for existing Multimodal Large Language Models (MLLMs). We identify two core challenges: \textit{entangled spatio-temporal alignment}, arising from coupling two heterogeneous sub-tasks within the same autoregressive output space, and \textit{dual-domain visual token redundancy}, where target objects exhibit simultaneous temporal and spatial sparsity, rendering the overwhelming majority of visual tokens irrelevant to the grounding query. To address these, we propose \textbf{Bridge-STG}, an end-to-end framework that decouples temporal and spatial localization while maintaining semantic coherence. While decoupling is the natural solution to this entanglement, it risks creating a semantic gap between the temporal MLLM and the spatial decoder. Bridge-STG resolves this through two pivotal designs: the \textbf{Spatio-Temporal Semantic Bridging (STSB)} mechanism with Explicit Temporal Alignment (ETA) distills the MLLM's temporal reasoning context into enriched bridging queries as a robust semantic interface; and the \textbf{Query-Guided Spatial Localization (QGSL)} module leverages these queries to drive a purpose-built spatial decoder with multi-layer interactive queries and positive/negative frame sampling, jointly eliminating dual-domain visual token redundancy. Extensive experiments across multiple benchmarks demonstrate that Bridge-STG achieves state-of-the-art performance among MLLM-based methods. Bridge-STG improves average m\_vIoU from $26.4$ to $34.3$ on VidSTG and demonstrates strong cross-task transfer across various fine-grained video understanding tasks under a unified multi-task training regime.

Related papers

• An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation [55.291219073365546]
  We introduce textbfMulti-temporal Referring (MTRS), a new task that aims to segment language-described temporal changes from multi-temporal images.<n>MTRS extends conventional referring segmentation and change detection by jointly requiring temporal correspondence reasoning, language grounding, and pixel-level mask prediction.<n>We show that MTRefSeg-R1 achieves strong and often superior performance compared with existing LVLM baselines.
  arXiv  Detail & Related papers  (2026-05-31T04:01:10Z)
• MASRA: MLLM-Assisted Semantic-Relational Consistent Alignment for Video Temporal Grounding [21.833707346184386]
  Video Temporal Grounding (VTG) faces a cross-modal semantic gap that often leads to background being incorrectly aligned with the query.<n>We propose MLLM-Assisted Semantic-Relational Consistent Alignment (MASRA), a training-time MLLM-based optimization framework for VTG.
  arXiv  Detail & Related papers  (2026-05-05T06:20:33Z)
• 1 + 1 > 2: Detector-Empowered Video Large Language Model for Spatio-Temporal Grounding and Reasoning [53.28271278708241]
  We present a Detector-Empowered Video LLM, short for DEViL.<n> DEViL couples a Video LLM with an open-vocabulary detector (OVD)<n>Unlike tokens that merely serve as spatial prompts or segmentor switches, the RST functions as both a control signal and a replacement for the OVD's text embedding.
  arXiv  Detail & Related papers  (2025-12-07T06:11:15Z)
• Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding [47.400649582392255]
  We use large language models (MLLMs) to explore a zero-shot solution in STVG.<n>We propose a MLLM-based zero-shot framework for STVG, which includes novel temporal-augmented assembling strategies.
  arXiv  Detail & Related papers  (2025-09-18T17:35:50Z)
• Identity-Preserving Text-to-Video Generation Guided by Simple yet Effective Spatial-Temporal Decoupled Representations [131.33758144860988]
  Identity-preserving text-to-video (IPT2V) generation aims to create high-fidelity videos with consistent human identity.<n>Current end-to-end frameworks suffer a critical spatial-temporal trade-off.<n>We propose a simple yet effective spatial-temporal decoupled framework that decomposes representations into spatial features for layouts and temporal features for motion dynamics.
  arXiv  Detail & Related papers  (2025-07-07T06:54:44Z)
• CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos [59.391265901911005]
  We propose CoT-RVS, a novel framework employing the zero-shot Chain-of-Thought (CoT) capability of MLLM to address complex challenges by temporal-semantic reasoning.<n>CoT-RVS analyzes the visible objects within a given frame that possibly match the language query (semantic), and chooses a corresponding for each object that can be observed effortlessly among all frames (temporal)<n>Our framework's training-free feature further allows its extension to process online video streams, where the CoT is used at test time to update the object of interest when a better target starts to emerge
  arXiv  Detail & Related papers  (2025-05-24T07:01:31Z)
• SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capability [58.46310813774538]
  Large language models (LMLMs) have made remarkable progress in either temporal or spatial localization.<n>However they struggle to perform-temporal video grounding.<n>This limitation stems from two major challenges.<n>We introduce SpaceLM, a MLLMVL endowed with temporal-temporal video grounding.
  arXiv  Detail & Related papers  (2025-03-18T07:40:36Z)
• LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding [29.42797944919497]
  We propose LLaVA-ST, a MLLM for fine-grained spatial-temporal multimodal understanding.<n>In LLaVA-ST, we propose Language-Aligned Positional Embedding, which embeds the coordinate special token into the visual space.<n>We also design the Spatial-Temporal Packer, which decouples the feature compression of temporal and spatial resolutions into two distinct point-to-region attention processing streams.
  arXiv  Detail & Related papers  (2025-01-14T17:58:12Z)
• Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation [62.275143240798236]
  Video semantic segmentation dataset has limited categories per video. Less than 10% of queries could be matched to receive meaningful gradient updates during VSS training. Our method achieves state-of-the-art performance on the latest challenging VSS benchmark VSPW without bells and whistles.
  arXiv  Detail & Related papers  (2023-09-14T20:31:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.