Fine-Grained Scene Graph Generation via Sample-Level Bias Prediction

• URL: http://arxiv.org/abs/2407.19259v1
• Date: Sat, 27 Jul 2024 13:49:06 GMT
• Title: Fine-Grained Scene Graph Generation via Sample-Level Bias Prediction
• Authors: Yansheng Li, Tingzhu Wang, Kang Wu, Linlin Wang, Xin Guo, Wenbin Wang,
• Abstract summary: We propose a novel Sample-Level Bias Prediction (SBP) method for fine-grained Scene Graph Generation (SGG) Firstly, we train a classic SGG model and construct a correction bias set. Then, we devise a Bias-Oriented Generative Adversarial Network (BGAN) that learns to predict the constructed correction biases.
• Score: 12.319354506916547
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scene Graph Generation (SGG) aims to explore the relationships between objects in images and obtain scene summary graphs, thereby better serving downstream tasks. However, the long-tailed problem has adversely affected the scene graph's quality. The predictions are dominated by coarse-grained relationships, lacking more informative fine-grained ones. The union region of one object pair (i.e., one sample) contains rich and dedicated contextual information, enabling the prediction of the sample-specific bias for refining the original relationship prediction. Therefore, we propose a novel Sample-Level Bias Prediction (SBP) method for fine-grained SGG (SBG). Firstly, we train a classic SGG model and construct a correction bias set by calculating the margin between the ground truth label and the predicted label with one classic SGG model. Then, we devise a Bias-Oriented Generative Adversarial Network (BGAN) that learns to predict the constructed correction biases, which can be utilized to correct the original predictions from coarse-grained relationships to fine-grained ones. The extensive experimental results on VG, GQA, and VG-1800 datasets demonstrate that our SBG outperforms the state-of-the-art methods in terms of Average@K across three mainstream SGG models: Motif, VCtree, and Transformer. Compared to dataset-level correction methods on VG, SBG shows a significant average improvement of 5.6%, 3.9%, and 3.2% on Average@K for tasks PredCls, SGCls, and SGDet, respectively. The code will be available at https://github.com/Zhuzi24/SBG.

Related papers

• Improving Scene Graph Generation with Relation Words' Debiasing in Vision-Language Models [6.8754535229258975]
  Scene Graph Generation (SGG) provides basic language representation of visual scenes. Part of test triplets are rare or even unseen during training, resulting in predictions. We propose using the SGG models with pretrained vision-language models (VLMs) to enhance representation.
  arXiv  Detail & Related papers  (2024-03-24T15:02:24Z)
• TD^2-Net: Toward Denoising and Debiasing for Dynamic Scene Graph Generation [76.24766055944554]
  We introduce a network named TD$2$-Net that aims at denoising and debiasing for dynamic SGG. TD$2$-Net outperforms the second-best competitors by 12.7 % on mean-Recall@10 for predicate classification.
  arXiv  Detail & Related papers  (2024-01-23T04:17:42Z)
• Informative Scene Graph Generation via Debiasing [111.36290856077584]
  Scene graph generation aims to detect visual relationship triplets, (subject, predicate, object) Due to biases in data, current models tend to predict common predicates. We propose DB-SGG, an effective framework based on debiasing but not the conventional distribution fitting.
  arXiv  Detail & Related papers  (2023-08-10T02:04:01Z)
• CAME: Context-aware Mixture-of-Experts for Unbiased Scene Graph Generation [10.724516317292926]
  We present a simple yet effective method called Context-Aware Mixture-of-Experts (CAME) to improve the model diversity and alleviate the biased scene graph generator. We have conducted extensive experiments on three tasks on the Visual Genome dataset to show that came achieved superior performance over previous methods.
  arXiv  Detail & Related papers  (2022-08-15T10:39:55Z)
• Adaptive Fine-Grained Predicates Learning for Scene Graph Generation [122.4588401267544]
  General Scene Graph Generation (SGG) models tend to predict head predicates and re-balancing strategies prefer tail categories. We propose an Adaptive Fine-Grained Predicates Learning (FGPL-A) which aims at differentiating hard-to-distinguish predicates for SGG. Our proposed model-agnostic strategy significantly boosts performance of benchmark models on VG-SGG and GQA-SGG datasets by up to 175% and 76% on Mean Recall@100, achieving new state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-07-11T03:37:57Z)
• Resistance Training using Prior Bias: toward Unbiased Scene Graph Generation [47.69807004675605]
  Scene Graph Generation (SGG) aims to build a structured representation of a scene using objects and pairwise relationships. We propose Resistance Training using Prior Bias (RTPB) for the scene graph generation. Our RTPB achieves an improvement of over 10% under the mean recall when applied to current SGG methods.
  arXiv  Detail & Related papers  (2022-01-18T07:48:55Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)
• Unbiased Scene Graph Generation from Biased Training [99.88125954889937]
  We present a novel SGG framework based on causal inference but not the conventional likelihood. We propose to draw the counterfactual causality from the trained graph to infer the effect from the bad bias. In particular, we use Total Direct Effect (TDE) as the proposed final predicate score for unbiased SGG.
  arXiv  Detail & Related papers  (2020-02-27T07:29:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.