A Paradigm Shift: Fully End-to-End Training for Temporal Sentence Grounding in Videos

• URL: http://arxiv.org/abs/2604.02860v1
• Date: Fri, 03 Apr 2026 08:26:12 GMT
• Title: A Paradigm Shift: Fully End-to-End Training for Temporal Sentence Grounding in Videos
• Abstract summary: Temporal sentence grounding in videos (TSGV) aims to localize a temporal segment that semantically corresponds to a sentence query from an untrimmed video.<n>Most current methods adopt pre-trained query-agnostic visual encoders for offline feature extraction, and the video backbones are frozen and not optimized for TSGV.<n>We propose a fully end-to-end paradigm that jointly optimize the video backbone and localization head.
• Score: 27.879461637513984
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Temporal sentence grounding in videos (TSGV) aims to localize a temporal segment that semantically corresponds to a sentence query from an untrimmed video. Most current methods adopt pre-trained query-agnostic visual encoders for offline feature extraction, and the video backbones are frozen and not optimized for TSGV. This leads to a task discrepancy issue for the video backbone trained for visual classification, but utilized for TSGV. To bridge this gap, we propose a fully end-to-end paradigm that jointly optimizes the video backbone and localization head. We first conduct an empirical study validating the effectiveness of end-to-end learning over frozen baselines across different model scales. Furthermore, we introduce a Sentence Conditioned Adapter (SCADA), which leverages sentence features to train a small portion of video backbone parameters adaptively. SCADA facilitates the deployment of deeper network backbones with reduced memory and significantly enhances visual representation by modulating feature maps through precise integration of linguistic embeddings. Experiments on two benchmarks show that our method outperforms state-of-the-art approaches. The code and models will be released.

Related papers

• DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding [11.584150931573738]
  Text-guided Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed video based on text queries.<n>Recent training-free VTG approaches mitigate this issue by directly matching pretrained vision-language representations.<n>We propose DSE-VTG, a framework that addresses both frame-wise visual encoding and fixed query embeddings.
  arXiv  Detail & Related papers  (2026-09-08T15:03:41Z)
• Learning from Streaming Video with Orthogonal Gradients [62.51504086522027]
  We address the challenge of representation learning from a continuous stream of video as input, in a self-supervised manner.<n>This differs from the standard approaches to video learning where videos are chopped and shuffled during training in order to create a non-redundant batch.<n>We demonstrate the drop in performance when moving from shuffled to sequential learning on three tasks.
  arXiv  Detail & Related papers  (2025-04-02T17:59:57Z)
• Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding [108.79026216923984]
  Video grounding aims to localize a-temporal section in a video corresponding to an input text query. This paper addresses a critical limitation in current video grounding methodologies by introducing an Open-Vocabulary Spatio-Temporal Video Grounding task.
  arXiv  Detail & Related papers  (2023-12-31T13:53:37Z)
• VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding [63.075626670943116]
  We introduce a cutting-edge framework, VaQuitA, designed to refine the synergy between video and textual information. At the data level, instead of sampling frames uniformly, we implement a sampling method guided by CLIP-score rankings. At the feature level, we integrate a trainable Video Perceiver alongside a Visual-Query Transformer.
  arXiv  Detail & Related papers  (2023-12-04T19:48:02Z)
• TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale [59.01246141215051]
  We analyze the factor that leads to degradation from the perspective of language supervision. We propose a tunable-free pre-training strategy to retain the generalization ability of the text encoder. We produce a series of models, dubbed TVTSv2, with up to one billion parameters.
  arXiv  Detail & Related papers  (2023-05-23T15:44:56Z)
• Transform-Equivariant Consistency Learning for Temporal Sentence Grounding [66.10949751429781]
  We introduce a novel Equivariant Consistency Regulation Learning framework to learn more discriminative representations for each video. Our motivation comes from that the temporal boundary of the query-guided activity should be consistently predicted. In particular, we devise a self-supervised consistency loss module to enhance the completeness and smoothness of the augmented video.
  arXiv  Detail & Related papers  (2023-05-06T19:29:28Z)
• Spatio-Temporal Crop Aggregation for Video Representation Learning [33.296154476701055]
  Our model builds long-range video features by learning from sets of video clip-level features extracted with a pre-trained backbone. We demonstrate that our video representation yields state-of-the-art performance with linear, non-linear, and $k$-NN probing on common action classification datasets.
  arXiv  Detail & Related papers  (2022-11-30T14:43:35Z)
• Frozen CLIP Models are Efficient Video Learners [86.73871814176795]
  Video recognition has been dominated by the end-to-end learning paradigm. Recent advances in Contrastive Vision-Language Pre-training pave the way for a new route for visual recognition tasks. We present Efficient Video Learning -- an efficient framework for directly training high-quality video recognition models.
  arXiv  Detail & Related papers  (2022-08-06T17:38:25Z)
• Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning [44.412145665354736]
  We introduce a novel contrastive action representation learning framework to learn frame-wise action representations. Inspired by the recent progress of self-supervised learning, we present a novel sequence contrastive loss (SCL) applied on two correlated views. Our approach also shows outstanding performance on video alignment and fine-grained frame retrieval tasks.
  arXiv  Detail & Related papers  (2022-03-28T17:59:54Z)
• End-to-End Dense Video Grounding via Parallel Regression [30.984657885692553]
  Video grounding aims to localize the corresponding video moment in an untrimmed video given a language query. We present an end-to-end parallel decoding paradigm by re-purposing a Transformer-alike architecture (PRVG) Thanks to its simplicity in design, our PRVG framework can be applied in different testing schemes.
  arXiv  Detail & Related papers  (2021-09-23T10:03:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.