Hierarchical Relationships: A New Perspective to Enhance Scene Graph Generation

• URL: http://arxiv.org/abs/2303.06842v5
• Date: Wed, 29 Nov 2023 05:32:33 GMT
• Title: Hierarchical Relationships: A New Perspective to Enhance Scene Graph Generation
• Authors: Bowen Jiang and Camillo J. Taylor
• Abstract summary: This paper presents a finding that leveraging the hierarchical structures among labels for relationships and objects can substantially improve the performance of scene graph generation systems. We introduce a Bayesian prediction head to jointly predict the super-category of relationships between a pair of object instances. Experiments on the Visual Genome dataset show its strong performance, particularly in predicate classifications and zero-shot settings.
• Score: 8.28849026314542
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a finding that leveraging the hierarchical structures among labels for relationships and objects can substantially improve the performance of scene graph generation systems. The focus of this work is to create an informative hierarchical structure that can divide object and relationship categories into disjoint super-categories in a systematic way. Specifically, we introduce a Bayesian prediction head to jointly predict the super-category of relationships between a pair of object instances, as well as the detailed relationship within that super-category simultaneously, facilitating more informative predictions. The resulting model exhibits the capability to produce a more extensive set of predicates beyond the dataset annotations, and to tackle the prevalent issue of low annotation quality. While our paper presents preliminary findings, experiments on the Visual Genome dataset show its strong performance, particularly in predicate classifications and zero-shot settings, that demonstrates the promise of our approach.

Related papers

• From Logits to Hierarchies: Hierarchical Clustering made Simple [16.132657141993548]
  We show that a lightweight procedure implemented on top of pre-trained non-hierarchical clustering models outperforms models designed specifically for hierarchical clustering. Our proposed approach is computationally efficient and applicable to any pre-trained clustering model that outputs logits, without requiring any fine-tuning.
  arXiv  Detail & Related papers  (2024-10-10T12:27:45Z)
• Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge [7.28830964611216]
  This work introduces an enhanced approach to generating scene graphs by both a relationship hierarchy and commonsense knowledge. We implement a robust commonsense validation pipeline that harnesses foundation models to critique the results from the scene graph prediction system. Experiments on Visual Genome and OpenImage V6 datasets demonstrate that the proposed modules can be seamlessly integrated as plug-and-play enhancements to existing scene graph generation algorithms.
  arXiv  Detail & Related papers  (2023-11-21T06:03:20Z)
• Graph-based Time Series Clustering for End-to-End Hierarchical Forecasting [18.069747511100132]
  Relationships among time series can be exploited as inductive biases in learning effective forecasting models. We propose a graph-based methodology to unify relational and hierarchical inductive biases.
  arXiv  Detail & Related papers  (2023-05-30T16:27:25Z)
• Hierarchical clustering with dot products recovers hidden tree structure [53.68551192799585]
  In this paper we offer a new perspective on the well established agglomerative clustering algorithm, focusing on recovery of hierarchical structure. We recommend a simple variant of the standard algorithm, in which clusters are merged by maximum average dot product and not, for example, by minimum distance or within-cluster variance. We demonstrate that the tree output by this algorithm provides a bona fide estimate of generative hierarchical structure in data, under a generic probabilistic graphical model.
  arXiv  Detail & Related papers  (2023-05-24T11:05:12Z)
• Prototype-based Embedding Network for Scene Graph Generation [105.97836135784794]
  Current Scene Graph Generation (SGG) methods explore contextual information to predict relationships among entity pairs. Due to the diverse visual appearance of numerous possible subject-object combinations, there is a large intra-class variation within each predicate category. Prototype-based Embedding Network (PE-Net) models entities/predicates with prototype-aligned compact and distinctive representations. PL is introduced to help PE-Net efficiently learn such entitypredicate matching, and Prototype Regularization (PR) is devised to relieve the ambiguous entity-predicate matching.
  arXiv  Detail & Related papers  (2023-03-13T13:30:59Z)
• Iterative Scene Graph Generation [55.893695946885174]
  Scene graph generation involves identifying object entities and their corresponding interaction predicates in a given image (or video) Existing approaches to scene graph generation assume certain factorization of the joint distribution to make the estimation iteration feasible. We propose a novel framework that addresses this limitation, as well as introduces dynamic conditioning on the image.
  arXiv  Detail & Related papers  (2022-07-27T10:37:29Z)
• Relation Regularized Scene Graph Generation [206.76762860019065]
  Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations. We propose a relation regularized network (R2-Net) which can predict whether there is a relationship between two objects. Our R2-Net can effectively refine object labels and generate scene graphs.
  arXiv  Detail & Related papers  (2022-02-22T11:36:49Z)
• Effective and Efficient Graph Learning for Multi-view Clustering [173.8313827799077]
  We propose an effective and efficient graph learning model for multi-view clustering. Our method exploits the view-similar between graphs of different views by the minimization of tensor Schatten p-norm. Our proposed algorithm is time-economical and obtains the stable results and scales well with the data size.
  arXiv  Detail & Related papers  (2021-08-15T13:14:28Z)
• Automated Concatenation of Embeddings for Structured Prediction [75.44925576268052]
  We propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks. We follow strategies in reinforcement learning to optimize the parameters of the controller and compute the reward based on the accuracy of a task model.
  arXiv  Detail & Related papers  (2020-10-10T14:03:20Z)
• Reasoning with Latent Structure Refinement for Document-Level Relation Extraction [20.308845516900426]
  We propose a novel model that empowers the relational reasoning across sentences by automatically inducing the latent document-level graph. Specifically, our model achieves an F1 score of 59.05 on a large-scale document-level dataset (DocRED)
  arXiv  Detail & Related papers  (2020-05-13T13:36:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.