BCTR: Bidirectional Conditioning Transformer for Scene Graph Generation

• URL: http://arxiv.org/abs/2407.18715v1
• Date: Fri, 26 Jul 2024 13:02:48 GMT
• Title: BCTR: Bidirectional Conditioning Transformer for Scene Graph Generation
• Authors: Peng Hao, Xiaobing Wang, Yingying Jiang, Hanchao Jia, Xiaoshuai Hao,
• Abstract summary: Scene Graph Generation (SGG) remains a challenging task due to its compositional property. Previous approaches improve prediction efficiency by learning in an end-to-end manner. We propose a novel bidirectional conditioning factorization for SGG, introducing efficient interaction between entities and predicates.
• Score: 4.977568882858193
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Scene Graph Generation (SGG) remains a challenging task due to its compositional property. Previous approaches improve prediction efficiency by learning in an end-to-end manner. However, these methods exhibit limited performance as they assume unidirectional conditioning between entities and predicates, leading to insufficient information interaction. To address this limitation, we propose a novel bidirectional conditioning factorization for SGG, introducing efficient interaction between entities and predicates. Specifically, we develop an end-to-end scene graph generation model, Bidirectional Conditioning Transformer (BCTR), to implement our factorization. BCTR consists of two key modules. First, the Bidirectional Conditioning Generator (BCG) facilitates multi-stage interactive feature augmentation between entities and predicates, enabling mutual benefits between the two predictions. Second, Random Feature Alignment (RFA) regularizes the feature space by distilling multi-modal knowledge from pre-trained models, enhancing BCTR's ability on tailed categories without relying on statistical priors. We conduct a series of experiments on Visual Genome and Open Image V6, demonstrating that BCTR achieves state-of-the-art performance on both benchmarks. The code will be available upon acceptance of the paper.

Related papers

• Multitask Fine-Tuning and Generative Adversarial Learning for Improved Auxiliary Classification [0.0]
  We implement a novel BERT architecture for multitask fine-tuning on three downstream tasks. Our model, Multitask BERT, incorporates layer sharing and a triplet architecture, custom sentence pair tokenization, loss pairing, and gradient surgery. We also apply generative adversarial learning to BERT, constructing a conditional generator model that maps from latent space to create fake embeddings.
  arXiv  Detail & Related papers  (2024-08-11T20:05:54Z)
• S^2Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in OR [50.435592120607815]
  Scene graph generation (SGG) of surgical procedures is crucial in enhancing holistically cognitive intelligence in the operating room (OR) Previous works have primarily relied on multi-stage learning, where the generated semantic scene graphs depend on intermediate processes with pose estimation and object detection. In this study, we introduce a novel single-stage bi-modal transformer framework for SGG in the OR, termed S2Former-OR.
  arXiv  Detail & Related papers  (2024-02-22T11:40:49Z)
• ALF: Adaptive Label Finetuning for Scene Graph Generation [116.59868289196157]
  Scene Graph Generation endeavors to predict the relationships between subjects and objects in a given image. Long-tail distribution of relations often leads to biased prediction on coarse labels, presenting a substantial hurdle in SGG. We introduce one-stage data transfer pipeline in SGG, termed Adaptive Label Finetuning (ALF), which eliminates the need for extra retraining sessions. ALF achieves a 16% improvement in mR@100 compared to the typical SGG method Motif, with only a 6% increase in calculation costs compared to the state-of-the-art method IETrans.
  arXiv  Detail & Related papers  (2023-12-29T01:37:27Z)
• Single-Stage Visual Relationship Learning using Conditional Queries [60.90880759475021]
  TraCQ is a new formulation for scene graph generation that avoids the multi-task learning problem and the entity pair distribution. We employ a DETR-based encoder-decoder conditional queries to significantly reduce the entity label space as well. Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats many state-of-the-art two-stage methods on the Visual Genome dataset.
  arXiv  Detail & Related papers  (2023-06-09T06:02:01Z)
• Transform-Equivariant Consistency Learning for Temporal Sentence Grounding [66.10949751429781]
  We introduce a novel Equivariant Consistency Regulation Learning framework to learn more discriminative representations for each video. Our motivation comes from that the temporal boundary of the query-guided activity should be consistently predicted. In particular, we devise a self-supervised consistency loss module to enhance the completeness and smoothness of the augmented video.
  arXiv  Detail & Related papers  (2023-05-06T19:29:28Z)
• On the Role of Bidirectionality in Language Model Pre-Training [85.14614350372004]
  We study the role of bidirectionality in next token prediction, text infilling, zero-shot priming and fine-tuning. We train models with up to 6.7B parameters, and find differences to remain consistent at scale.
  arXiv  Detail & Related papers  (2022-05-24T02:25:05Z)
• Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation [62.96628432641806]
  Scene Graph Generation aims to first encode the visual contents within the given image and then parse them into a compact summary graph. We first present a novel Stacked Hybrid-Attention network, which facilitates the intra-modal refinement as well as the inter-modal interaction. We then devise an innovative Group Collaborative Learning strategy to optimize the decoder.
  arXiv  Detail & Related papers  (2022-03-18T09:14:13Z)
• Temporal Action Proposal Generation with Transformers [25.66256889923748]
  This paper intuitively presents a unified temporal action proposal generation framework with original Transformers. The Boundary Transformer captures long-term temporal dependencies to predict precise boundary information. The Proposal Transformer learns the rich inter-proposal relationships for reliable confidence evaluation.
  arXiv  Detail & Related papers  (2021-05-25T16:22:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.