HL-Net: Heterophily Learning Network for Scene Graph Generation

• URL: http://arxiv.org/abs/2205.01316v2
• Date: Wed, 4 May 2022 01:04:20 GMT
• Title: HL-Net: Heterophily Learning Network for Scene Graph Generation
• Authors: Xin Lin, Changxing Ding, Yibing Zhan, Zijian Li, Dacheng Tao
• Abstract summary: We propose a novel Heterophily Learning Network (HL-Net) to explore the homophily and heterophily between objects/relationships in scene graphs. HL-Net comprises the following 1) an adaptive reweighting transformer module, which adaptively integrates the information from different layers to exploit both the heterophily and homophily in objects. We conducted extensive experiments on two public datasets: Visual Genome (VG) and Open Images (OI)
• Score: 90.2766568914452
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Scene graph generation (SGG) aims to detect objects and predict their pairwise relationships within an image. Current SGG methods typically utilize graph neural networks (GNNs) to acquire context information between objects/relationships. Despite their effectiveness, however, current SGG methods only assume scene graph homophily while ignoring heterophily. Accordingly, in this paper, we propose a novel Heterophily Learning Network (HL-Net) to comprehensively explore the homophily and heterophily between objects/relationships in scene graphs. More specifically, HL-Net comprises the following 1) an adaptive reweighting transformer module, which adaptively integrates the information from different layers to exploit both the heterophily and homophily in objects; 2) a relationship feature propagation module that efficiently explores the connections between relationships by considering heterophily in order to refine the relationship representation; 3) a heterophily-aware message-passing scheme to further distinguish the heterophily and homophily between objects/relationships, thereby facilitating improved message passing in graphs. We conducted extensive experiments on two public datasets: Visual Genome (VG) and Open Images (OI). The experimental results demonstrate the superiority of our proposed HL-Net over existing state-of-the-art approaches. In more detail, HL-Net outperforms the second-best competitors by 2.1$\%$ on the VG dataset for scene graph classification and 1.2$\%$ on the IO dataset for the final score. Code is available at https://github.com/siml3/HL-Net.

Related papers

• Semantic Scene Graph Generation Based on an Edge Dual Scene Graph and Message Passing Neural Network [3.9280441311534653]
  Scene graph generation (SGG) captures the relationships between objects in an image and creates a structured graph-based representation. Existing SGG methods have a limited ability to accurately predict detailed relationships. A new approach to the modeling multiobject relationships, called edge dual scene graph generation (EdgeSGG), is proposed herein.
  arXiv  Detail & Related papers  (2023-11-02T12:36:52Z)
• Unbiased Heterogeneous Scene Graph Generation with Relation-aware Message Passing Neural Network [9.779600950401315]
  We propose an unbiased heterogeneous scene graph generation (HetSGG) framework that captures relation-aware context. We devise a novel message passing layer, called relation-aware message passing neural network (RMP), that aggregates the contextual information of an image.
  arXiv  Detail & Related papers  (2022-12-01T11:25:36Z)
• RU-Net: Regularized Unrolling Network for Scene Graph Generation [92.95032610978511]
  Scene graph generation (SGG) aims to detect objects and predict the relationships between each pair of objects. Existing SGG methods usually suffer from several issues, including 1) ambiguous object representations, and 2) low diversity in relationship predictions. We propose a regularized unrolling network (RU-Net) to address both problems.
  arXiv  Detail & Related papers  (2022-05-03T04:21:15Z)
• Heterogeneous Graph Neural Networks using Self-supervised Reciprocally Contrastive Learning [102.9138736545956]
  Heterogeneous graph neural network (HGNN) is a very popular technique for the modeling and analysis of heterogeneous graphs. We develop for the first time a novel and robust heterogeneous graph contrastive learning approach, namely HGCL, which introduces two views on respective guidance of node attributes and graph topologies. In this new approach, we adopt distinct but most suitable attribute and topology fusion mechanisms in the two views, which are conducive to mining relevant information in attributes and topologies separately.
  arXiv  Detail & Related papers  (2022-04-30T12:57:02Z)
• Relation Regularized Scene Graph Generation [206.76762860019065]
  Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations. We propose a relation regularized network (R2-Net) which can predict whether there is a relationship between two objects. Our R2-Net can effectively refine object labels and generate scene graphs.
  arXiv  Detail & Related papers  (2022-02-22T11:36:49Z)
• Hyper-relationship Learning Network for Scene Graph Generation [95.6796681398668]
  We propose a hyper-relationship learning network, termed HLN, for scene graph generation. We evaluate HLN on the most popular SGG dataset, i.e., the Visual Genome dataset. For example, the proposed HLN improves the recall per relationship from 11.3% to 13.1%, and maintains the recall per image from 19.8% to 34.9%.
  arXiv  Detail & Related papers  (2022-02-15T09:26:16Z)
• Graph Neural Networks for Graphs with Heterophily: A Survey [98.45621222357397]
  We provide a comprehensive review of graph neural networks (GNNs) for heterophilic graphs. Specifically, we propose a systematic taxonomy that essentially governs existing heterophilic GNN models. We discuss the correlation between graph heterophily and various graph research domains, aiming to facilitate the development of more effective GNNs.
  arXiv  Detail & Related papers  (2022-02-14T23:07:47Z)
• Fully Convolutional Scene Graph Generation [30.194961716870186]
  This paper presents a fully convolutional scene graph generation (FCSGG) model that detects objects and relations simultaneously. FCSGG encodes objects as bounding box center points, and relationships as 2D vector fields which are named as Relation Affinity Fields (RAFs) FCSGG achieves highly competitive results on recall and zero-shot recall with significantly reduced inference time.
  arXiv  Detail & Related papers  (2021-03-30T05:25:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.