TD^2-Net: Toward Denoising and Debiasing for Dynamic Scene Graph Generation

• URL: http://arxiv.org/abs/2401.12479v1
• Date: Tue, 23 Jan 2024 04:17:42 GMT
• Title: TD^2-Net: Toward Denoising and Debiasing for Dynamic Scene Graph Generation
• Authors: Xin Lin, Chong Shi, Yibing Zhan, Zuopeng Yang, Yaqi Wu, Dacheng Tao
• Abstract summary: We introduce a network named TD$2$-Net that aims at denoising and debiasing for dynamic SGG. TD$2$-Net outperforms the second-best competitors by 12.7 % on mean-Recall@10 for predicate classification.
• Score: 76.24766055944554
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Dynamic scene graph generation (SGG) focuses on detecting objects in a video and determining their pairwise relationships. Existing dynamic SGG methods usually suffer from several issues, including 1) Contextual noise, as some frames might contain occluded and blurred objects. 2) Label bias, primarily due to the high imbalance between a few positive relationship samples and numerous negative ones. Additionally, the distribution of relationships exhibits a long-tailed pattern. To address the above problems, in this paper, we introduce a network named TD$^2$-Net that aims at denoising and debiasing for dynamic SGG. Specifically, we first propose a denoising spatio-temporal transformer module that enhances object representation with robust contextual information. This is achieved by designing a differentiable Top-K object selector that utilizes the gumbel-softmax sampling strategy to select the relevant neighborhood for each object. Second, we introduce an asymmetrical reweighting loss to relieve the issue of label bias. This loss function integrates asymmetry focusing factors and the volume of samples to adjust the weights assigned to individual samples. Systematic experimental results demonstrate the superiority of our proposed TD$^2$-Net over existing state-of-the-art approaches on Action Genome databases. In more detail, TD$^2$-Net outperforms the second-best competitors by 12.7 \% on mean-Recall@10 for predicate classification.

Related papers

• Fine-Grained Scene Graph Generation via Sample-Level Bias Prediction [12.319354506916547]
  We propose a novel Sample-Level Bias Prediction (SBP) method for fine-grained Scene Graph Generation (SGG) Firstly, we train a classic SGG model and construct a correction bias set. Then, we devise a Bias-Oriented Generative Adversarial Network (BGAN) that learns to predict the constructed correction biases.
  arXiv  Detail & Related papers  (2024-07-27T13:49:06Z)
• FloCoDe: Unbiased Dynamic Scene Graph Generation with Temporal Consistency and Correlation Debiasing [14.50214193838818]
  FloCoDe: Flow-aware Temporal and Correlation Debiasing with uncertainty attenuation for unbiased dynamic scene graphs. We propose correlation debiasing and a correlation-based loss to learn unbiased relation representations for long-tailed classes.
  arXiv  Detail & Related papers  (2023-10-24T14:59:51Z)
• Informative Scene Graph Generation via Debiasing [124.71164256146342]
  Scene graph generation aims to detect visual relationship triplets, (subject, predicate, object) Due to biases in data, current models tend to predict common predicates. We propose DB-SGG, an effective framework based on debiasing but not the conventional distribution fitting.
  arXiv  Detail & Related papers  (2023-08-10T02:04:01Z)
• Delving into Identify-Emphasize Paradigm for Combating Unknown Bias [52.76758938921129]
  We propose an effective bias-conflicting scoring method (ECS) to boost the identification accuracy. We also propose gradient alignment (GA) to balance the contributions of the mined bias-aligned and bias-conflicting samples. Experiments are conducted on multiple datasets in various settings, demonstrating that the proposed solution can mitigate the impact of unknown biases.
  arXiv  Detail & Related papers  (2023-02-22T14:50:24Z)
• NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation [65.78472854070316]
  We propose a novel NoIsy label CorrEction and Sample Training strategy for SGG: NICEST. NICE first detects noisy samples and then reassigns them more high-quality soft predicate labels. NICEST can be seamlessly incorporated into any SGG architecture to boost its performance on different predicate categories.
  arXiv  Detail & Related papers  (2022-07-27T06:25:47Z)
• RU-Net: Regularized Unrolling Network for Scene Graph Generation [92.95032610978511]
  Scene graph generation (SGG) aims to detect objects and predict the relationships between each pair of objects. Existing SGG methods usually suffer from several issues, including 1) ambiguous object representations, and 2) low diversity in relationship predictions. We propose a regularized unrolling network (RU-Net) to address both problems.
  arXiv  Detail & Related papers  (2022-05-03T04:21:15Z)
• Instance-Level Relative Saliency Ranking with Graph Reasoning [126.09138829920627]
  We present a novel unified model to segment salient instances and infer relative saliency rank order. A novel loss function is also proposed to effectively train the saliency ranking branch. experimental results demonstrate that our proposed model is more effective than previous methods.
  arXiv  Detail & Related papers  (2021-07-08T13:10:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.