DMC$^3$: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering

• URL: http://arxiv.org/abs/2510.20285v1
• Date: Thu, 23 Oct 2025 07:15:18 GMT
• Title: DMC$^3$: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering
• Abstract summary: Egocentric Video Question Answering (Egocentric VideoQA) plays an important role in egocentric video understanding.<n>We propose a Dual-Modal Counterfactual Contrastive Construction framework, which contains an egocentric videoqa baseline, a counterfactual sample construction module and a counterfactual sample-involved contrastive optimization.<n>Our method achieve 52.51% and 46.04% on the textitnormal and textitindirect splits of EgoTaskQA, and 13.2% on QAEGO4D.
• Score: 66.45476128172444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Egocentric Video Question Answering (Egocentric VideoQA) plays an important role in egocentric video understanding, which refers to answering questions based on first-person videos. Although existing methods have made progress through the paradigm of pre-training and fine-tuning, they ignore the unique challenges posed by the first-person perspective, such as understanding multiple events and recognizing hand-object interactions. To deal with these challenges, we propose a Dual-Modal Counterfactual Contrastive Construction (DMC$^3$) framework, which contains an egocentric videoqa baseline, a counterfactual sample construction module and a counterfactual sample-involved contrastive optimization. Specifically, We first develop a counterfactual sample construction module to generate positive and negative samples for textual and visual modalities through event description paraphrasing and core interaction mining, respectively. Then, We feed these samples together with the original samples into the baseline. Finally, in the counterfactual sample-involved contrastive optimization module, we apply contrastive loss to minimize the distance between the original sample features and the positive sample features, while maximizing the distance from the negative samples. Experiments show that our method achieve 52.51\% and 46.04\% on the \textit{normal} and \textit{indirect} splits of EgoTaskQA, and 13.2\% on QAEGO4D, both reaching the state-of-the-art performance.

Related papers

• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• Beyond Isolated Facts: Synthesizing Narrative and Grounded Supervision for VideoQA [37.679936989592996]
  We introduce a framework to synthesize richer supervisory signals.<n>We propose two complementary strategies: Question-Based Paraphrasing (QBP) and Question-Based Captioning (QBC)
  arXiv  Detail & Related papers  (2025-09-29T08:28:44Z)
• Weakly Supervised Temporal Sentence Grounding via Positive Sample Mining [64.25904019001449]
  The task of weakly supervised temporal sentence grounding (WSTSG) aims to detect temporal intervals corresponding to a language description from untrimmed videos with only video-level video-language correspondence.<n>For an anchor sample, most existing approaches generate negative samples either from other videos or within the same video for contrastive learning.<n>We propose Positive Sample Mining (PSM), a novel framework that mines positive samples from the training set to provide more discriminative supervision.
  arXiv  Detail & Related papers  (2025-05-10T08:03:00Z)
• Enhanced Partially Relevant Video Retrieval through Inter- and Intra-Sample Analysis with Coherence Prediction [18.24629930062925]
  Partially Relevant Video Retrieval aims to retrieve the target video that is partially relevant to a text query.<n>Existing methods coarsely align paired videos and text queries to construct the semantic space.<n>We propose a novel PRVR framework to systematically exploit inter-sample correlation and intra-sample redundancy.
  arXiv  Detail & Related papers  (2025-04-28T09:52:46Z)
• Beat: Bi-directional One-to-Many Embedding Alignment for Text-based Person Retrieval [66.61856014573742]
  Text-based person retrieval (TPR) is a challenging task that involves retrieving a specific individual based on a textual description. Previous methods have attempted to align text and image samples in a modal-shared space. We propose an effective bi-directional one-to-many embedding paradigm that offers a clear optimization direction for each sample.
  arXiv  Detail & Related papers  (2024-06-09T03:06:55Z)
• Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning [54.61762276179205]
  We propose a novel contrastive learning approach, MMBS, for building robust VQA models by Making the Most of Biased Samples. Specifically, we construct positive samples for contrastive learning by eliminating the information related to spurious correlation from the original training samples. We validate our contributions by achieving competitive performance on the OOD dataset VQA-CP v2 while preserving robust performance on the ID dataset VQA v2.
  arXiv  Detail & Related papers  (2022-10-10T11:05:21Z)
• Consensus Synergizes with Memory: A Simple Approach for Anomaly Segmentation in Urban Scenes [132.16748656557013]
  Anomaly segmentation is a crucial task for safety-critical applications, such as autonomous driving in urban scenes. We propose a novel and simple approach named Consensus Synergizes with Memory (CosMe) to address this challenge. Experimental results on several urban scene anomaly segmentation datasets show that CosMe outperforms previous approaches by large margins.
  arXiv  Detail & Related papers  (2021-11-24T10:01:20Z)
• Paired Examples as Indirect Supervision in Latent Decision Models [109.76417071249945]
  We introduce a way to leverage paired examples that provide stronger cues for learning latent decisions. We apply our method to improve compositional question answering using neural module networks on the DROP dataset.
  arXiv  Detail & Related papers  (2021-04-05T03:58:30Z)
• Doubly Contrastive Deep Clustering [135.7001508427597]
  We present a novel Doubly Contrastive Deep Clustering (DCDC) framework, which constructs contrastive loss over both sample and class views. Specifically, for the sample view, we set the class distribution of the original sample and its augmented version as positive sample pairs. For the class view, we build the positive and negative pairs from the sample distribution of the class. In this way, two contrastive losses successfully constrain the clustering results of mini-batch samples in both sample and class level.
  arXiv  Detail & Related papers  (2021-03-09T15:15:32Z)
• Neural Passage Retrieval with Improved Negative Contrast [21.415627458949356]
  We explore four negative sampling strategies that complement the straightforward random sampling of negatives, typically used to train dual encoder models. Our retrieval-based strategies are based on the semantic similarity and the lexical overlap between questions and passages. Our results establish a new state-of-the-art level of performance on two of the open-domain question answering datasets.
  arXiv  Detail & Related papers  (2020-10-23T16:45:06Z)
• Self-supervised pre-training and contrastive representation learning for multiple-choice video QA [39.78914328623504]
  Video Question Answering (Video QA) requires fine-grained understanding of both video and language modalities to answer the given questions. We propose novel training schemes for multiple-choice video question answering with a self-supervised pre-training stage and a supervised contrastive learning in the main stage as an auxiliary learning. We evaluate our proposed model on highly competitive benchmark datasets related to multiple-choice video QA: TVQA, TVQA+, and DramaQA.
  arXiv  Detail & Related papers  (2020-09-17T03:37:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.