Boosting Temporal Sentence Grounding via Causal Inference

• URL: http://arxiv.org/abs/2507.04958v1
• Date: Mon, 07 Jul 2025 13:01:06 GMT
• Title: Boosting Temporal Sentence Grounding via Causal Inference
• Authors: Kefan Tang, Lihuo He, Jisheng Dang, Xinbo Gao,
• Abstract summary: Temporal Sentence Grounding aims to identify relevant moments in an untrimmed video that semantically correspond to a given textual query.<n>These spurious correlations arise from two primary factors: (1) inherent biases in the textual data, such as frequent co-occurrences of specific verbs or phrases, and (2) the model's tendency to overfit to salient or repetitive patterns in video content.<n>We propose a novel TSG framework, causal intervention and counterfactual reasoning that utilize causal inference to eliminate spurious correlations and enhance the model's robustness.
• Score: 48.04297516212874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Temporal Sentence Grounding (TSG) aims to identify relevant moments in an untrimmed video that semantically correspond to a given textual query. Despite existing studies having made substantial progress, they often overlook the issue of spurious correlations between video and textual queries. These spurious correlations arise from two primary factors: (1) inherent biases in the textual data, such as frequent co-occurrences of specific verbs or phrases, and (2) the model's tendency to overfit to salient or repetitive patterns in video content. Such biases mislead the model into associating textual cues with incorrect visual moments, resulting in unreliable predictions and poor generalization to out-of-distribution examples. To overcome these limitations, we propose a novel TSG framework, causal intervention and counterfactual reasoning that utilizes causal inference to eliminate spurious correlations and enhance the model's robustness. Specifically, we first formulate the TSG task from a causal perspective with a structural causal model. Then, to address unobserved confounders reflecting textual biases toward specific verbs or phrases, a textual causal intervention is proposed, utilizing do-calculus to estimate the causal effects. Furthermore, visual counterfactual reasoning is performed by constructing a counterfactual scenario that focuses solely on video features, excluding the query and fused multi-modal features. This allows us to debias the model by isolating and removing the influence of the video from the overall effect. Experiments on public datasets demonstrate the superiority of the proposed method. The code is available at https://github.com/Tangkfan/CICR.

Related papers

• Bias-Conflict Sample Synthesis and Adversarial Removal Debias Strategy for Temporal Sentence Grounding in Video [67.24316233946381]
  Temporal Sentence Grounding in Video (TSGV) is troubled by dataset bias issue. We propose the bias-conflict sample synthesis and adversarial removal debias strategy (BSSARD)
  arXiv  Detail & Related papers  (2024-01-15T09:59:43Z)
• Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Intervention [72.12974259966592]
  We present a unique and systematic study of a temporal bias due to frame length discrepancy between training and test sets of trimmed video clips. We propose a causal debiasing approach and perform extensive experiments and ablation studies on the Epic-Kitchens-100, YouCook2, and MSR-VTT datasets.
  arXiv  Detail & Related papers  (2023-09-17T15:58:27Z)
• Causalainer: Causal Explainer for Automatic Video Summarization [77.36225634727221]
  In many application scenarios, improper video summarization can have a large impact. Modeling explainability is a key concern. A Causal Explainer, dubbed Causalainer, is proposed to address this issue.
  arXiv  Detail & Related papers  (2023-04-30T11:42:06Z)
• Learning Sample Importance for Cross-Scenario Video Temporal Grounding [30.82619216537177]
  The paper investigates some superficial biases specific to the temporal grounding task. We propose a novel method called Debiased Temporal Language Localizer (DebiasTLL) to prevent the model from naively memorizing the biases. We evaluate the proposed model in cross-scenario temporal grounding, where the train / test data are heterogeneously sourced.
  arXiv  Detail & Related papers  (2022-01-08T15:41:38Z)
• Towards Debiasing Temporal Sentence Grounding in Video [59.42702544312366]
  temporal sentence grounding in video (TSGV) task is to locate a temporal moment from an untrimmed video, to match a language query. Without considering bias in moment annotations, many models tend to capture statistical regularities of the moment annotations. We propose two debiasing strategies, data debiasing and model debiasing, to "force" a TSGV model to capture cross-modal interactions.
  arXiv  Detail & Related papers  (2021-11-08T08:18:25Z)
• Interventional Video Grounding with Dual Contrastive Learning [16.0734337895897]
  Video grounding aims to localize a moment from an untrimmed video for a given textual query. We propose a novel paradigm from the perspective of causal inference to uncover the causality behind the model and data. We also introduce a dual contrastive learning approach to better align the text and video.
  arXiv  Detail & Related papers  (2021-06-21T12:11:28Z)
• Deconfounded Video Moment Retrieval with Causal Intervention [80.90604360072831]
  We tackle the task of video moment retrieval (VMR), which aims to localize a specific moment in a video according to a textual query. Existing methods primarily model the matching relationship between query and moment by complex cross-modal interactions. We propose a causality-inspired VMR framework that builds structural causal model to capture the true effect of query and video content on the prediction.
  arXiv  Detail & Related papers  (2021-06-03T01:33:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.