Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge

• URL: http://arxiv.org/abs/2311.12889v2
• Date: Tue, 16 Jul 2024 04:39:05 GMT
• Title: Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge
• Authors: Bowen Jiang, Zhijun Zhuang, Shreyas S. Shivakumar, Camillo J. Taylor,
• Abstract summary: This work introduces an enhanced approach to generating scene graphs by both a relationship hierarchy and commonsense knowledge. We implement a robust commonsense validation pipeline that harnesses foundation models to critique the results from the scene graph prediction system. Experiments on Visual Genome and OpenImage V6 datasets demonstrate that the proposed modules can be seamlessly integrated as plug-and-play enhancements to existing scene graph generation algorithms.
• Score: 7.28830964611216
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work introduces an enhanced approach to generating scene graphs by incorporating both a relationship hierarchy and commonsense knowledge. Specifically, we begin by proposing a hierarchical relation head that exploits an informative hierarchical structure. It jointly predicts the relation super-category between object pairs in an image, along with detailed relations under each super-category. Following this, we implement a robust commonsense validation pipeline that harnesses foundation models to critique the results from the scene graph prediction system, removing nonsensical predicates even with a small language-only model. Extensive experiments on Visual Genome and OpenImage V6 datasets demonstrate that the proposed modules can be seamlessly integrated as plug-and-play enhancements to existing scene graph generation algorithms. The results show significant improvements with an extensive set of reasonable predictions beyond dataset annotations. Codes are available at https://github.com/bowen-upenn/scene_graph_commonsense.

Related papers

• Hierarchical Relationships: A New Perspective to Enhance Scene Graph Generation [8.28849026314542]
  This paper presents a finding that leveraging the hierarchical structures among labels for relationships and objects can substantially improve the performance of scene graph generation systems. We introduce a Bayesian prediction head to jointly predict the super-category of relationships between a pair of object instances. Experiments on the Visual Genome dataset show its strong performance, particularly in predicate classifications and zero-shot settings.
  arXiv  Detail & Related papers  (2023-03-13T04:16:42Z)
• High-order Multi-view Clustering for Generic Data [15.764819403555512]
  Graph-based multi-view clustering has achieved better performance than most non-graph approaches. We introduce an approach called high-order multi-view clustering (HMvC) to explore the topology structure information of generic data.
  arXiv  Detail & Related papers  (2022-09-22T07:49:38Z)
• Scene Graph Modification as Incremental Structure Expanding [61.84291817776118]
  We focus on scene graph modification (SGM), where the system is required to learn how to update an existing scene graph based on a natural language query. We frame SGM as a graph expansion task by introducing the incremental structure expanding (ISE) We construct a challenging dataset that contains more complicated queries and larger scene graphs than existing datasets.
  arXiv  Detail & Related papers  (2022-09-15T16:26:14Z)
• Iterative Scene Graph Generation [55.893695946885174]
  Scene graph generation involves identifying object entities and their corresponding interaction predicates in a given image (or video) Existing approaches to scene graph generation assume certain factorization of the joint distribution to make the estimation iteration feasible. We propose a novel framework that addresses this limitation, as well as introduces dynamic conditioning on the image.
  arXiv  Detail & Related papers  (2022-07-27T10:37:29Z)
• Relation Regularized Scene Graph Generation [206.76762860019065]
  Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations. We propose a relation regularized network (R2-Net) which can predict whether there is a relationship between two objects. Our R2-Net can effectively refine object labels and generate scene graphs.
  arXiv  Detail & Related papers  (2022-02-22T11:36:49Z)
• One-shot Scene Graph Generation [130.57405850346836]
  We propose Multiple Structured Knowledge (Relational Knowledgesense Knowledge) for the one-shot scene graph generation task. Our method significantly outperforms existing state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2022-02-22T11:32:59Z)
• RelTR: Relation Transformer for Scene Graph Generation [34.1193503312965]
  We propose an end-to-end scene graph generation model RelTR with an encoder-decoder architecture. The model infers a fixed-size set of triplets subject-predicate-object using different types of attention mechanisms. Experiments on the Visual Genome and Open Images V6 datasets demonstrate the superior performance and fast inference of our model.
  arXiv  Detail & Related papers  (2022-01-27T11:53:41Z)
• A Robust and Generalized Framework for Adversarial Graph Embedding [73.37228022428663]
  We propose a robust framework for adversarial graph embedding, named AGE. AGE generates the fake neighbor nodes as the enhanced negative samples from the implicit distribution. Based on this framework, we propose three models to handle three types of graph data.
  arXiv  Detail & Related papers  (2021-05-22T07:05:48Z)
• Bridging Knowledge Graphs to Generate Scene Graphs [49.69377653925448]
  We propose a novel graph-based neural network that iteratively propagates information between the two graphs, as well as within each of them. Our Graph Bridging Network, GB-Net, successively infers edges and nodes, allowing to simultaneously exploit and refine the rich, heterogeneous structure of the interconnected scene and commonsense graphs.
  arXiv  Detail & Related papers  (2020-01-07T23:35:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.