Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification

• URL: http://arxiv.org/abs/2606.29023v1
• Date: Sat, 27 Jun 2026 17:42:32 GMT
• Title: Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification
• Abstract summary: temporal-temporal grounding in long videos requires precise temporal localization and robust object tracking conditioned on natural-language queries.<n>We propose a practical pipeline that shifts from frame-level to second-level tracking and performs cross-second annotations to preserve continuity while reducing sequence length.
• Score: 11.171207895220578
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spatio-temporal grounding in long videos requires precise temporal localization and robust object tracking conditioned on natural-language queries. While recent vision-language models (VLMs) show strong reasoning ability, directly applying frame-by-frame inference to long sequences is computationally expensive and unstable. We propose a practical pipeline that shifts from frame-level to second-level tracking and performs cross-second smoothing to preserve continuity while reducing sequence length. To improve reasoning supervision, we synthesize chain-of-thought style trajectories using advanced multimodal models for temporal localization and target selection, and replace generated spatio-temporal coordinates with ground-truth annotations to avoid noisy supervision. We further optimize the policy with reinforcement learning using a verifier based on $t\_\mathrm{IoU}+mv\_\mathrm{IoU}$. Experiments across multiple FPS settings show that our method achieves a strong trade-off between efficiency and localization quality.

Related papers

• Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding [80.10744910189997]
  multimodal large language models typically serialize trajectories autoregressively.<n>We introduce Parallel Tube Decoding (PTD), a generative formulation that decomposes grounding into a temporal block.<n>On VidSTG, PTD Tube reduces Completion by 79x and increases spatial decoding by 92x over standard autoregressive decoding.
  arXiv  Detail & Related papers  (2026-08-28T11:05:30Z)
• ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding [23.883675818190188]
  S-Temporal Video Grounding aims to retrieve the visual trajectory of a specific object from a video stream as described by a natural language expression.<n>Most advanced methods struggle to balance global context modeling with precise boundary localization.<n>We present a novel coarse-to-fine framework that decouples the STVG task into a global-temporal scan and a local boundary focus.
  arXiv  Detail & Related papers  (2026-07-15T03:50:22Z)
• UCM: Unifying Camera Control and Memory with Time-aware Positional Encoding Warping for World Models [54.564740558030245]
  We present UCM, a novel framework that unifies long-term memory and precise camera control via a time-aware positional encoding warping mechanism.<n>We also introduce a scalable data curation strategy utilizing point-cloud-based rendering to simulate scene revisiting.
  arXiv  Detail & Related papers  (2026-02-26T12:54:46Z)
• Agentic Spatio-Temporal Grounding via Collaborative Reasoning [80.83158605034465]
  Temporal Video Grounding aims to retrieve thetemporal tube of a target object or person in a video given a text query.<n>We propose the Agentic Spatio-Temporal Grounder (ASTG) framework for the task of STVG towards an open-world and training-free scenario.<n>Specifically, two specialized agents SRA (Spatial Reasoning Agent) and TRA (Temporal Reasoning Agent) constructed leveraging on modern Multimoal Large Language Models (MLLMs)<n>Experiments on popular benchmarks demonstrate the superiority of the proposed approach where it outperforms existing weakly-supervised and zero-shot approaches by a margin
  arXiv  Detail & Related papers  (2026-02-10T10:16:27Z)
• Exploring Reliable Spatiotemporal Dependencies for Efficient Visual Tracking [9.64398631601942]
  STDTrack is a framework that pioneers the integration of reliabletemporal dependencies into lightweight trackers.<n>We introduce a temporally propagatingtemporal token to guide per-frame feature extraction.<n>We develop a multi-scale prediction head to dynamically adapt to objects of different sizes.
  arXiv  Detail & Related papers  (2026-01-14T02:22:05Z)
• Multi-modal Deepfake Detection and Localization with FPN-Transformer [21.022230340898556]
  We introduce a multi-modal deepfake detection and localization framework based on a Feature Pyramid-Transformer (FPN-Transformer)<n>A multi-scale feature pyramid is constructed through R-TLM blocks with localized attention mechanisms, enabling joint analysis of cross-context temporal dependencies.<n>We evaluate our approach on the test set of the IJCAI'25 DDL-AV benchmark, showing a good performance with a final score of 0.7535.
  arXiv  Detail & Related papers  (2025-11-11T09:33:39Z)
• Towards Robust and Generalizable Continuous Space-Time Video Super-Resolution with Events [71.2439653098351]
  Continuous space-time video super-STVSR has garnered increasing interest for its capability to reconstruct high-resolution and high-frame-rate videos at arbitrary temporal scales.<n>We present EvEnhancer, a novel approach that marries unique properties of high temporal and high dynamic range encapsulated in event streams.<n>Our method achieves state-of-the-art performance on both synthetic and real-world datasets, while maintaining generalizability at OOD scales.
  arXiv  Detail & Related papers  (2025-10-04T15:23:07Z)
• PatchTraj: Unified Time-Frequency Representation Learning via Dynamic Patches for Trajectory Prediction [14.48846131633279]
  We propose a dynamic patch-based framework that integrates time-frequency joint modeling for trajectory prediction.<n> Specifically, we decompose the trajectory into raw time sequences and frequency components, and employ dynamic patch partitioning to perform multi-scale segmentation.<n>The resulting enhanced embeddings exhibit strong expressive power, enabling accurate predictions even when using a vanilla architecture.
  arXiv  Detail & Related papers  (2025-07-25T09:55:33Z)
• FDDet: Frequency-Decoupling for Boundary Refinement in Temporal Action Detection [4.015022008487465]
  Large-scale pre-trained video encoders tend to introduce background clutter and irrelevant semantics, leading to context confusion and boundaries.<n>We propose a frequency-aware decoupling network that improves action discriminability by filtering out noisy semantics captured by pre-trained models.<n>Our method achieves state-of-the-art performance on temporal action detection benchmarks.
  arXiv  Detail & Related papers  (2025-04-01T10:57:37Z)
• SuperFlow++: Enhanced Spatiotemporal Consistency for Cross-Modal Data Pretraining [62.433137130087445]
  SuperFlow++ is a novel framework that integrates pretraining and downstream tasks using consecutive camera pairs.<n>We show that SuperFlow++ outperforms state-of-the-art methods across diverse tasks and driving conditions.<n>With strong generalizability and computational efficiency, SuperFlow++ establishes a new benchmark for data-efficient LiDAR-based perception in autonomous driving.
  arXiv  Detail & Related papers  (2025-03-25T17:59:57Z)
• Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking [53.33637391723555]
  We propose a unified multimodal spatial-temporal tracking approach named STTrack.<n>In contrast to previous paradigms, we introduced a temporal state generator (TSG) that continuously generates a sequence of tokens containing multimodal temporal information.<n>These temporal information tokens are used to guide the localization of the target in the next time state, establish long-range contextual relationships between video frames, and capture the temporal trajectory of the target.
  arXiv  Detail & Related papers  (2024-12-20T09:10:17Z)
• Multi-step Temporal Modeling for UAV Tracking [14.687636301587045]
  We introduce MT-Track, a streamlined and efficient multi-step temporal modeling framework for enhanced UAV tracking. We unveil a unique temporal correlation module that dynamically assesses the interplay between the template and search region features. We propose a mutual transformer module to refine the correlation maps of historical and current frames by modeling the temporal knowledge in the tracking sequence.
  arXiv  Detail & Related papers  (2024-03-07T09:48:13Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.