SG-Shuffle: Multi-aspect Shuffle Transformer for Scene Graph Generation

• URL: http://arxiv.org/abs/2211.04773v1
• Date: Wed, 9 Nov 2022 10:00:45 GMT
• Title: SG-Shuffle: Multi-aspect Shuffle Transformer for Scene Graph Generation
• Abstract summary: Scene Graph Generation (SGG) serves a comprehensive representation of the images for human understanding and visual understanding tasks. Due to the long tail bias problem of the object and predicate labels in the available annotated data, the scene graph generated from current methodologies can be biased toward common, non-informative relationship labels. In this work, we proposed the SG-Shuffle pipeline for scene graph generation with 3 components: 1) Parallel Transformer, which learns to predict object relationships in a more exclusive manner by grouping relationship labels into groups of similar purpose; 2) Shuffle Transformer, which learns to select the final relationship labels from the category-specific feature
• Score: 4.319987790757823
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scene Graph Generation (SGG) serves a comprehensive representation of the images for human understanding as well as visual understanding tasks. Due to the long tail bias problem of the object and predicate labels in the available annotated data, the scene graph generated from current methodologies can be biased toward common, non-informative relationship labels. Relationship can sometimes be non-mutually exclusive, which can be described from multiple perspectives like geometrical relationships or semantic relationships, making it even more challenging to predict the most suitable relationship label. In this work, we proposed the SG-Shuffle pipeline for scene graph generation with 3 components: 1) Parallel Transformer Encoder, which learns to predict object relationships in a more exclusive manner by grouping relationship labels into groups of similar purpose; 2) Shuffle Transformer, which learns to select the final relationship labels from the category-specific feature generated in the previous step; and 3) Weighted CE loss, used to alleviate the training bias caused by the imbalanced dataset.

Related papers

• Can We Build Scene Graphs, Not Classify Them? FlowSG: Progressive Image-Conditioned Scene Graph Generation with Flow Matching [35.40374108008262]
  Scene Graph Generation (SGG) unifies object localization and visual relationship reasoning by predicting boxes and subject-predicate-object triples.<n>We propose FlowSG, which recasts SGG as continuous-time transport on a hybrid discrete-continuous state.<n>Experiments on VG and PSG under closed- and open-vocabulary protocols show consistent gains in predicate R/mR and graph-level metrics.
  arXiv  Detail & Related papers  (2026-04-18T01:48:43Z)
• SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance [46.77060502803466]
  We introduce the Scene Graph Adapter(SG-Adapter), leveraging the structured representation of scene graphs to rectify inaccuracies in the original text embeddings. The SG-Adapter's explicit and non-fully connected graph representation greatly improves the fully connected, transformer-based text representations.
  arXiv  Detail & Related papers  (2024-05-24T08:00:46Z)
• Relation Rectification in Diffusion Model [64.84686527988809]
  We introduce a novel task termed Relation Rectification, aiming to refine the model to accurately represent a given relationship it initially fails to generate. We propose an innovative solution utilizing a Heterogeneous Graph Convolutional Network (HGCN) The lightweight HGCN adjusts the text embeddings generated by the text encoder, ensuring the accurate reflection of the textual relation in the embedding space.
  arXiv  Detail & Related papers  (2024-03-29T15:54:36Z)
• DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation [13.058196732927135]
  Scene graph generation aims to capture detailed spatial and semantic relationships between objects in an image. Existing Transformer-based methods either employ distinct queries for objects and predicates or utilize holistic queries for relation triplets. We present a new Transformer-based method, called DSGG, that views scene graph detection as a direct graph prediction problem.
  arXiv  Detail & Related papers  (2024-03-21T23:43:30Z)
• Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation [55.429541407920304]
  Recognizing the predicate between subject and object pairs is imbalanced and multi-label in nature. Recent state-of-the-art methods predominantly focus on the most frequently occurring predicate classes. We introduce a multi-label meta-learning framework to deal with the biased predicate distribution.
  arXiv  Detail & Related papers  (2023-06-16T18:14:23Z)
• CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion [83.30168660888913]
  We present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model.
  arXiv  Detail & Related papers  (2023-05-25T17:39:13Z)
• Panoptic Scene Graph Generation [41.534209967051645]
  panoptic scene graph generation (PSG) is a new problem task that requires the model to generate a more comprehensive scene graph representation. A high-quality PSG dataset contains 49k well-annotated overlapping images from COCO and Visual Genome.
  arXiv  Detail & Related papers  (2022-07-22T17:59:53Z)
• Graph Attention Transformer Network for Multi-Label Image Classification [50.0297353509294]
  We propose a general framework for multi-label image classification that can effectively mine complex inter-label relationships. Our proposed methods can achieve state-of-the-art performance on three datasets.
  arXiv  Detail & Related papers  (2022-03-08T12:39:05Z)
• Relation Regularized Scene Graph Generation [206.76762860019065]
  Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations. We propose a relation regularized network (R2-Net) which can predict whether there is a relationship between two objects. Our R2-Net can effectively refine object labels and generate scene graphs.
  arXiv  Detail & Related papers  (2022-02-22T11:36:49Z)
• Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation [48.21846438269506]
  Scene graph generation (SGG) aims to capture a wide variety of interactions between pairs of objects. Existing SGG methods fail to acquire complex reasoning about visual and textual correlations due to various biases in training data. We propose a novel framework for SGG training that exploits relation labels based on their informativeness.
  arXiv  Detail & Related papers  (2021-11-26T14:34:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.