Scenes and Surroundings: Scene Graph Generation using Relation Transformer

• URL: http://arxiv.org/abs/2107.05448v1
• Date: Mon, 12 Jul 2021 14:22:20 GMT
• Title: Scenes and Surroundings: Scene Graph Generation using Relation Transformer
• Authors: Rajat Koner, Poulami Sinhamahapatra, Volker Tresp
• Abstract summary: This work proposes a novel local-context aware architecture named relation transformer. Our hierarchical multi-head attention-based approach efficiently captures contextual dependencies between objects and predicts their relationships. In comparison to state-of-the-art approaches, we have achieved an overall mean textbf4.85% improvement.
• Score: 13.146732454123326
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Identifying objects in an image and their mutual relationships as a scene graph leads to a deep understanding of image content. Despite the recent advancement in deep learning, the detection and labeling of visual object relationships remain a challenging task. This work proposes a novel local-context aware architecture named relation transformer, which exploits complex global objects to object and object to edge (relation) interactions. Our hierarchical multi-head attention-based approach efficiently captures contextual dependencies between objects and predicts their relationships. In comparison to state-of-the-art approaches, we have achieved an overall mean \textbf{4.85\%} improvement and a new benchmark across all the scene graph generation tasks on the Visual Genome dataset.

Related papers

• Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality [50.48859793121308]
  Contrastively trained vision-language models have achieved remarkable progress in vision and language representation learning. Recent research has highlighted severe limitations in their ability to perform compositional reasoning over objects, attributes, and relations.
  arXiv  Detail & Related papers  (2023-05-23T08:28:38Z)
• Grounding Scene Graphs on Natural Images via Visio-Lingual Message Passing [17.63475613154152]
  This paper presents a framework for jointly grounding objects that follow certain semantic relationship constraints in a scene graph. A scene graph is an efficient and structured way to represent all the objects and their semantic relationships in the image.
  arXiv  Detail & Related papers  (2022-11-03T16:46:46Z)
• Iterative Scene Graph Generation [55.893695946885174]
  Scene graph generation involves identifying object entities and their corresponding interaction predicates in a given image (or video) Existing approaches to scene graph generation assume certain factorization of the joint distribution to make the estimation iteration feasible. We propose a novel framework that addresses this limitation, as well as introduces dynamic conditioning on the image.
  arXiv  Detail & Related papers  (2022-07-27T10:37:29Z)
• Relationformer: A Unified Framework for Image-to-Graph Generation [18.832626244362075]
  This work proposes a unified one-stage transformer-based framework, namely Relationformer, that jointly predicts objects and their relations. We leverage direct set-based object prediction and incorporate the interaction among the objects to learn an object-relation representation jointly. We achieve state-of-the-art performance on multiple, diverse and multi-domain datasets.
  arXiv  Detail & Related papers  (2022-03-19T00:36:59Z)
• Transformer-based Dual Relation Graph for Multi-label Image Recognition [56.12543717723385]
  We propose a novel Transformer-based Dual Relation learning framework. We explore two aspects of correlation, i.e., structural relation graph and semantic relation graph. Our approach achieves new state-of-the-art on two popular multi-label recognition benchmarks.
  arXiv  Detail & Related papers  (2021-10-10T07:14:52Z)
• Unconditional Scene Graph Generation [72.53624470737712]
  We develop a deep auto-regressive model called SceneGraphGen which can learn the probability distribution over labelled and directed graphs. We show that the scene graphs generated by SceneGraphGen are diverse and follow the semantic patterns of real-world scenes.
  arXiv  Detail & Related papers  (2021-08-12T17:57:16Z)
• Segmentation-grounded Scene Graph Generation [47.34166260639392]
  We propose a framework for pixel-level segmentation-grounded scene graph generation. Our framework is agnostic to the underlying scene graph generation method. It is learned in a multi-task manner with both target and auxiliary datasets.
  arXiv  Detail & Related papers  (2021-04-29T08:54:08Z)
• Exploiting Relationship for Complex-scene Image Generation [43.022978211274065]
  This work explores relationship-aware complex-scene image generation, where multiple objects are inter-related as a scene graph. We propose three major updates in the generation framework. First, reasonable spatial layouts are inferred by jointly considering the semantics and relationships among objects. Second, since the relations between objects significantly influence an object's appearance, we design a relation-guided generator to generate objects reflecting their relationships. Third, a novel scene graph discriminator is proposed to guarantee the consistency between the generated image and the input scene graph.
  arXiv  Detail & Related papers  (2021-04-01T09:21:39Z)
• ORD: Object Relationship Discovery for Visual Dialogue Generation [60.471670447176656]
  We propose an object relationship discovery (ORD) framework to preserve the object interactions for visual dialogue generation. A hierarchical graph convolutional network (HierGCN) is proposed to retain the object nodes and neighbour relationships locally, and then refines the object-object connections globally. Experiments have proved that the proposed method can significantly improve the quality of dialogue by utilising the contextual information of visual relationships.
  arXiv  Detail & Related papers  (2020-06-15T12:25:40Z)
• Object-Centric Image Generation from Layouts [93.10217725729468]
  We develop a layout-to-image-generation method to generate complex scenes with multiple objects. Our method learns representations of the spatial relationships between objects in the scene, which lead to our model's improved layout-fidelity. We introduce SceneFID, an object-centric adaptation of the popular Fr'echet Inception Distance metric, that is better suited for multi-object images.
  arXiv  Detail & Related papers  (2020-03-16T21:40:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.