Energy-Based Learning for Scene Graph Generation
- URL: http://arxiv.org/abs/2103.02221v1
- Date: Wed, 3 Mar 2021 07:11:23 GMT
- Title: Energy-Based Learning for Scene Graph Generation
- Authors: Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan
Eledath, Gerard Medioni, Leonid Sigal
- Abstract summary: We introduce a novel energy-based learning framework for generating scene graphs.
The proposed formulation allows for efficiently incorporating the structure of scene graphs in the output space.
We use the proposed framework to train existing state-of-the-art models and obtain a significant performance improvement.
- Score: 26.500496033477127
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Traditional scene graph generation methods are trained using cross-entropy
losses that treat objects and relationships as independent entities. Such a
formulation, however, ignores the structure in the output space, in an
inherently structured prediction problem. In this work, we introduce a novel
energy-based learning framework for generating scene graphs. The proposed
formulation allows for efficiently incorporating the structure of scene graphs
in the output space. This additional constraint in the learning framework acts
as an inductive bias and allows models to learn efficiently from a small number
of labels. We use the proposed energy-based framework to train existing
state-of-the-art models and obtain a significant performance improvement, of up
to 21% and 27%, on the Visual Genome and GQA benchmark datasets, respectively.
Furthermore, we showcase the learning efficiency of the proposed framework by
demonstrating superior performance in the zero- and few-shot settings where
data is scarce.
Related papers
- Generative Modelling of Structurally Constrained Graphs [44.28728853270471]
We present ConStruct, a novel framework that allows for hard-constraining graph diffusion models to incorporate specific properties, such as planarity or acyclicity.
Our approach ensures that the sampled graphs remain within the domain of graphs that verify the specified property throughout the entire trajectory in both the forward and reverse processes.
arXiv Detail & Related papers (2024-06-25T07:54:32Z) - A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
We introduce a feature-centric pretraining perspective by treating graph structure as a prior.
Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks.
GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
arXiv Detail & Related papers (2024-06-19T22:30:08Z) - A structure-aware framework for learning device placements on computation graphs [15.282882425920064]
We propose a novel framework for the task of device placement, relying on smaller graphs extracted from the OpenVINO toolkit.
The framework consists of five steps, including graph coarsening, node representation learning and policy optimization.
We demonstrate the flexibility and effectiveness of our approach through multiple experiments with three benchmark models.
arXiv Detail & Related papers (2024-05-23T05:29:29Z) - GraphGLOW: Universal and Generalizable Structure Learning for Graph
Neural Networks [72.01829954658889]
This paper introduces the mathematical definition of this novel problem setting.
We devise a general framework that coordinates a single graph-shared structure learner and multiple graph-specific GNNs.
The well-trained structure learner can directly produce adaptive structures for unseen target graphs without any fine-tuning.
arXiv Detail & Related papers (2023-06-20T03:33:22Z) - Improving Knowledge Graph Entity Alignment with Graph Augmentation [11.1094009195297]
Entity alignment (EA) which links equivalent entities across different knowledge graphs (KGs) plays a crucial role in knowledge fusion.
In recent years, graph neural networks (GNNs) have been successfully applied in many embedding-based EA methods.
We propose graph augmentation to create two graph views for margin-based alignment learning and contrastive entity representation learning.
arXiv Detail & Related papers (2023-04-28T01:22:47Z) - Localized Contrastive Learning on Graphs [110.54606263711385]
We introduce a simple yet effective contrastive model named Localized Graph Contrastive Learning (Local-GCL)
In spite of its simplicity, Local-GCL achieves quite competitive performance in self-supervised node representation learning tasks on graphs with various scales and properties.
arXiv Detail & Related papers (2022-12-08T23:36:00Z) - GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text
Generation [3.593955557310285]
Recent improvements in KG-to-text generation are due to auxiliary pre-training tasks designed to give the fine-tune task a boost in performance.
Here, we demonstrate that by fusing graph-aware elements into existing pre-trained language models, we are able to outperform state-of-the-art models and close the gap imposed by additional pre-training tasks.
arXiv Detail & Related papers (2022-04-13T23:53:37Z) - Towards Unsupervised Deep Graph Structure Learning [67.58720734177325]
We propose an unsupervised graph structure learning paradigm, where the learned graph topology is optimized by data itself without any external guidance.
Specifically, we generate a learning target from the original data as an "anchor graph", and use a contrastive loss to maximize the agreement between the anchor graph and the learned graph.
arXiv Detail & Related papers (2022-01-17T11:57:29Z) - Effective and Efficient Graph Learning for Multi-view Clustering [173.8313827799077]
We propose an effective and efficient graph learning model for multi-view clustering.
Our method exploits the view-similar between graphs of different views by the minimization of tensor Schatten p-norm.
Our proposed algorithm is time-economical and obtains the stable results and scales well with the data size.
arXiv Detail & Related papers (2021-08-15T13:14:28Z) - Model-Agnostic Graph Regularization for Few-Shot Learning [60.64531995451357]
We present a comprehensive study on graph embedded few-shot learning.
We introduce a graph regularization approach that allows a deeper understanding of the impact of incorporating graph information between labels.
Our approach improves the performance of strong base learners by up to 2% on Mini-ImageNet and 6.7% on ImageNet-FS.
arXiv Detail & Related papers (2021-02-14T05:28:13Z) - Neural Stochastic Block Model & Scalable Community-Based Graph Learning [8.00785050036369]
This paper proposes a scalable community-based neural framework for graph learning.
The framework learns the graph topology through the task of community detection and link prediction.
We look into two particular applications, the graph alignment and the anomalous correlation detection.
arXiv Detail & Related papers (2020-05-16T03:28:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.