Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation

• URL: http://arxiv.org/abs/2208.03763v1
• Date: Sun, 7 Aug 2022 16:19:19 GMT
• Title: Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation
• Authors: Lin Li, Long Chen, Hanrong Shi, Wenxiao Wang, Jian Shao, Yi Yang, Jun Xiao
• Abstract summary: We propose a novel model-agnostic Label Semantic Knowledge Distillation (LS-KD) for unbiased Scene Graph Generation (SGG) LS-KD dynamically generates a soft label for each subject-object instance by fusing a predicted Label Semantic Distribution (LSD) with its original one-hot target label.
• Score: 34.20922091969159
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Scene Graph Generation (SGG) task aims to detect all the objects and their pairwise visual relationships in a given image. Although SGG has achieved remarkable progress over the last few years, almost all existing SGG models follow the same training paradigm: they treat both object and predicate classification in SGG as a single-label classification problem, and the ground-truths are one-hot target labels. However, this prevalent training paradigm has overlooked two characteristics of current SGG datasets: 1) For positive samples, some specific subject-object instances may have multiple reasonable predicates. 2) For negative samples, there are numerous missing annotations. Regardless of the two characteristics, SGG models are easy to be confused and make wrong predictions. To this end, we propose a novel model-agnostic Label Semantic Knowledge Distillation (LS-KD) for unbiased SGG. Specifically, LS-KD dynamically generates a soft label for each subject-object instance by fusing a predicted Label Semantic Distribution (LSD) with its original one-hot target label. LSD reflects the correlations between this instance and multiple predicate categories. Meanwhile, we propose two different strategies to predict LSD: iterative self-KD and synchronous self-KD. Extensive ablations and results on three SGG tasks have attested to the superiority and generality of our proposed LS-KD, which can consistently achieve decent trade-off performance between different predicate categories.

Related papers

• Semantic Diversity-aware Prototype-based Learning for Unbiased Scene Graph Generation [21.772806350802203]
  In scene graph generation (SGG) datasets, each subject-object pair is annotated with a single predicate. Existing SGG models are trained to predict the one and only predicate for each pair. This in turn results in the SGG models to overlook the semantic diversity that may exist in a predicate.
  arXiv  Detail & Related papers  (2024-07-22T05:53:46Z)
• Adaptive Self-training Framework for Fine-grained Scene Graph Generation [29.37568710952893]
  Scene graph generation (SGG) models have suffered from inherent problems regarding the benchmark datasets. We introduce a Self-Training framework for SGG (ST-SGG) that assigns pseudo-labels for unannotated triplets. Our experiments verify the effectiveness of ST-SGG on various SGG models.
  arXiv  Detail & Related papers  (2024-01-18T08:10:34Z)
• Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention [69.36723767339001]
  Scene Graph Generation (SGG) offers a structured representation critical in many computer vision applications. We propose a unified framework named OvSGTR towards fully open vocabulary SGG from a holistic view. For the more challenging settings of relation-involved open vocabulary SGG, the proposed approach integrates relation-aware pretraining.
  arXiv  Detail & Related papers  (2023-11-18T06:49:17Z)
• Informative Scene Graph Generation via Debiasing [124.71164256146342]
  Scene graph generation aims to detect visual relationship triplets, (subject, predicate, object) Due to biases in data, current models tend to predict common predicates. We propose DB-SGG, an effective framework based on debiasing but not the conventional distribution fitting.
  arXiv  Detail & Related papers  (2023-08-10T02:04:01Z)
• Decomposed Prototype Learning for Few-Shot Scene Graph Generation [28.796734816086065]
  We focus on a new promising task of scene graph generation (SGG): few-shot SGG (FSSGG) FSSGG encourages models to be able to quickly transfer previous knowledge and recognize novel predicates with only a few examples. We propose a novel Decomposed Prototype Learning (DPL)
  arXiv  Detail & Related papers  (2023-03-20T04:54:26Z)
• Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning [84.39787427288525]
  Scene graph generation (SGG) is a fundamental task aimed at detecting visual relations between objects in an image. We introduce open-vocabulary scene graph generation, a novel, realistic and challenging setting in which a model is trained on a set of base object classes. Our method can support inference over completely unseen object classes, which existing methods are incapable of handling.
  arXiv  Detail & Related papers  (2022-08-17T09:05:38Z)
• NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation [65.78472854070316]
  We propose a novel NoIsy label CorrEction and Sample Training strategy for SGG: NICEST. NICE first detects noisy samples and then reassigns them more high-quality soft predicate labels. NICEST can be seamlessly incorporated into any SGG architecture to boost its performance on different predicate categories.
  arXiv  Detail & Related papers  (2022-07-27T06:25:47Z)
• Adaptive Fine-Grained Predicates Learning for Scene Graph Generation [122.4588401267544]
  General Scene Graph Generation (SGG) models tend to predict head predicates and re-balancing strategies prefer tail categories. We propose an Adaptive Fine-Grained Predicates Learning (FGPL-A) which aims at differentiating hard-to-distinguish predicates for SGG. Our proposed model-agnostic strategy significantly boosts performance of benchmark models on VG-SGG and GQA-SGG datasets by up to 175% and 76% on Mean Recall@100, achieving new state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-07-11T03:37:57Z)
• The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation [33.45310571580091]
  We propose a novel model-agnostic NoIsy label CorrEction strategy for unbiased SGG models. NICE can not only detect noisy samples but also reassign more high-quality predicate labels to them. NICE consists of three components: negative Noisy Sample Detection (Neg-NSD), positive NSD (Pos-NSD), and Noisy Sample Correction (NSC)
  arXiv  Detail & Related papers  (2022-06-07T05:03:57Z)
• Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation [48.21846438269506]
  Scene graph generation (SGG) aims to capture a wide variety of interactions between pairs of objects. Existing SGG methods fail to acquire complex reasoning about visual and textual correlations due to various biases in training data. We propose a novel framework for SGG training that exploits relation labels based on their informativeness.
  arXiv  Detail & Related papers  (2021-11-26T14:34:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.