Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency

• URL: http://arxiv.org/abs/2405.12648v1
• Date: Tue, 21 May 2024 09:56:48 GMT
• Title: Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency
• Authors: Hyeongjin Kim, Sangwon Kim, Dasom Ahn, Jong Taek Lee, Byoung Chul Ko,
• Abstract summary: Scene graph generation (SGG) represents the relationships between objects in an image as a graph structure. Previous studies have failed to reflect the co-occurrence of objects during SGG generation. We propose CooK, which reflects the Co-occurrence Knowledge between objects, and the learnable term frequency-inverse document frequency.
• Score: 3.351553095054309
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Scene graph generation (SGG) is an important task in image understanding because it represents the relationships between objects in an image as a graph structure, making it possible to understand the semantic relationships between objects intuitively. Previous SGG studies used a message-passing neural networks (MPNN) to update features, which can effectively reflect information about surrounding objects. However, these studies have failed to reflect the co-occurrence of objects during SGG generation. In addition, they only addressed the long-tail problem of the training dataset from the perspectives of sampling and learning methods. To address these two problems, we propose CooK, which reflects the Co-occurrence Knowledge between objects, and the learnable term frequency-inverse document frequency (TF-l-IDF) to solve the long-tail problem. We applied the proposed model to the SGG benchmark dataset, and the results showed a performance improvement of up to 3.8% compared with existing state-of-the-art models in SGGen subtask. The proposed method exhibits generalization ability from the results obtained, showing uniform performance improvement for all MPNN models.

Related papers

• AUG: A New Dataset and An Efficient Model for Aerial Image Urban Scene Graph Generation [40.149652254414185]
  This paper constructs and releases an aerial image urban scene graph generation (AUG) dataset. Images from the AUG dataset are captured with the low-attitude overhead view. To avoid the local context being overwhelmed in the complex aerial urban scene, this paper proposes one new locality-preserving graph convolutional network (LPG)
  arXiv  Detail & Related papers  (2024-04-11T14:29:30Z)
• EGTR: Extracting Graph from Transformer for Scene Graph Generation [5.935927309154952]
  Scene Graph Generation (SGG) is a challenging task of detecting objects and predicting relationships between objects. We propose a lightweight one-stage SGG model that extracts the relation graph from the various relationships learned in the multi-head self-attention layers of the DETR decoder. We demonstrate the effectiveness and efficiency of our method for the Visual Genome and Open Image V6 datasets.
  arXiv  Detail & Related papers  (2024-04-02T16:20:02Z)
• Semantic Scene Graph Generation Based on an Edge Dual Scene Graph and Message Passing Neural Network [3.9280441311534653]
  Scene graph generation (SGG) captures the relationships between objects in an image and creates a structured graph-based representation. Existing SGG methods have a limited ability to accurately predict detailed relationships. A new approach to the modeling multiobject relationships, called edge dual scene graph generation (EdgeSGG), is proposed herein.
  arXiv  Detail & Related papers  (2023-11-02T12:36:52Z)
• Adaptive Visual Scene Understanding: Incremental Scene Graph Generation [20.255178648494756]
  Scene graph generation (SGG) involves analyzing images to extract meaningful information about objects and their relationships. To address the lack of continual learning methodologies in SGG, we introduce the comprehensive Continual ScenE Graph Generation dataset.
  arXiv  Detail & Related papers  (2023-10-02T21:02:23Z)
• Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning [84.39787427288525]
  Scene graph generation (SGG) is a fundamental task aimed at detecting visual relations between objects in an image. We introduce open-vocabulary scene graph generation, a novel, realistic and challenging setting in which a model is trained on a set of base object classes. Our method can support inference over completely unseen object classes, which existing methods are incapable of handling.
  arXiv  Detail & Related papers  (2022-08-17T09:05:38Z)
• HL-Net: Heterophily Learning Network for Scene Graph Generation [90.2766568914452]
  We propose a novel Heterophily Learning Network (HL-Net) to explore the homophily and heterophily between objects/relationships in scene graphs. HL-Net comprises the following 1) an adaptive reweighting transformer module, which adaptively integrates the information from different layers to exploit both the heterophily and homophily in objects. We conducted extensive experiments on two public datasets: Visual Genome (VG) and Open Images (OI)
  arXiv  Detail & Related papers  (2022-05-03T06:00:29Z)
• Relation Regularized Scene Graph Generation [206.76762860019065]
  Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations. We propose a relation regularized network (R2-Net) which can predict whether there is a relationship between two objects. Our R2-Net can effectively refine object labels and generate scene graphs.
  arXiv  Detail & Related papers  (2022-02-22T11:36:49Z)
• Semantic Compositional Learning for Low-shot Scene Graph Generation [122.51930904132685]
  Many scene graph generation (SGG) models solely use the limited annotated relation triples for training. We propose a novel semantic compositional learning strategy that makes it possible to construct additional, realistic relation triples. For three recent SGG models, adding our strategy improves their performance by close to 50%, and all of them substantially exceed the current state-of-the-art.
  arXiv  Detail & Related papers  (2021-08-19T10:13:55Z)
• Unified Graph Structured Models for Video Understanding [93.72081456202672]
  We propose a message passing graph neural network that explicitly models relational-temporal relations. We show how our method is able to more effectively model relationships between relevant entities in the scene.
  arXiv  Detail & Related papers  (2021-03-29T14:37:35Z)
• Pairwise Relation Learning for Semi-supervised Gland Segmentation [90.45303394358493]
  We propose a pairwise relation-based semi-supervised (PRS2) model for gland segmentation on histology images. This model consists of a segmentation network (S-Net) and a pairwise relation network (PR-Net) We evaluate our model against five recent methods on the GlaS dataset and three recent methods on the CRAG dataset.
  arXiv  Detail & Related papers  (2020-08-06T15:02:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.