Fine-Grained is Too Coarse: A Novel Data-Centric Approach for Efficient Scene Graph Generation

• URL: http://arxiv.org/abs/2305.18668v2
• Date: Mon, 25 Sep 2023 12:35:00 GMT
• Title: Fine-Grained is Too Coarse: A Novel Data-Centric Approach for Efficient Scene Graph Generation
• Authors: Neau Ma\"elic, Paulo E. Santos, Anne-Gwenn Bosser and C\'edric Buche
• Abstract summary: We introduce the task of Efficient Scene Graph Generation (SGG) that prioritizes the generation of relevant relations. We present a new dataset, VG150-curated, based on the annotations of the popular Visual Genome dataset. We show through a set of experiments that this dataset contains more high-quality and diverse annotations than the one usually use in SGG.
• Score: 0.7851536646859476
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Learning to compose visual relationships from raw images in the form of scene graphs is a highly challenging task due to contextual dependencies, but it is essential in computer vision applications that depend on scene understanding. However, no current approaches in Scene Graph Generation (SGG) aim at providing useful graphs for downstream tasks. Instead, the main focus has primarily been on the task of unbiasing the data distribution for predicting more fine-grained relations. That being said, all fine-grained relations are not equally relevant and at least a part of them are of no use for real-world applications. In this work, we introduce the task of Efficient SGG that prioritizes the generation of relevant relations, facilitating the use of Scene Graphs in downstream tasks such as Image Generation. To support further approaches, we present a new dataset, VG150-curated, based on the annotations of the popular Visual Genome dataset. We show through a set of experiments that this dataset contains more high-quality and diverse annotations than the one usually use in SGG. Finally, we show the efficiency of this dataset in the task of Image Generation from Scene Graphs.

Related papers

• DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation [13.058196732927135]
  Scene graph generation aims to capture detailed spatial and semantic relationships between objects in an image. Existing Transformer-based methods either employ distinct queries for objects and predicates or utilize holistic queries for relation triplets. We present a new Transformer-based method, called DSGG, that views scene graph detection as a direct graph prediction problem.
  arXiv  Detail & Related papers  (2024-03-21T23:43:30Z)
• G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering [61.93058781222079]
  We develop a flexible question-answering framework targeting real-world textual graphs. We introduce the first retrieval-augmented generation (RAG) approach for general textual graphs. G-Retriever performs RAG over a graph by formulating this task as a Prize-Collecting Steiner Tree optimization problem.
  arXiv  Detail & Related papers  (2024-02-12T13:13:04Z)
• SelfGraphVQA: A Self-Supervised Graph Neural Network for Scene-based Question Answering [0.0]
  Scene graphs have emerged as a useful tool for multimodal image analysis. Current methods that utilize idealized annotated scene graphs struggle to generalize when using predicted scene graphs extracted from images. Our approach extracts a scene graph from an input image using a pre-trained scene graph generator.
  arXiv  Detail & Related papers  (2023-10-03T07:14:53Z)
• Learnable Graph Matching: A Practical Paradigm for Data Association [74.28753343714858]
  We propose a general learnable graph matching method to address these issues. Our method achieves state-of-the-art performance on several MOT datasets. For image matching, our method outperforms state-of-the-art methods on a popular indoor dataset, ScanNet.
  arXiv  Detail & Related papers  (2023-03-27T17:39:00Z)
• Location-Free Scene Graph Generation [45.366540803729386]
  Scene Graph Generation (SGG) is a visual understanding task, aiming to describe a scene as a graph of entities and their relationships with each other. Existing works rely on location labels in form of bounding boxes or segmentation masks, increasing annotation costs and limiting dataset expansion. We break this dependency and introduce location-free scene graph generation (LF-SGG) This new task aims at predicting instances of entities, as well as their relationships, without the explicit calculation of their spatial localization.
  arXiv  Detail & Related papers  (2023-03-20T08:57:45Z)
• Image Semantic Relation Generation [0.76146285961466]
  Scene graphs can distil complex image information and correct the bias of visual models using semantic-level relations. In this work, we introduce image semantic relation generation (ISRG), a simple but effective image-to-text model.
  arXiv  Detail & Related papers  (2022-10-19T16:15:19Z)
• Scene Graph Modification as Incremental Structure Expanding [61.84291817776118]
  We focus on scene graph modification (SGM), where the system is required to learn how to update an existing scene graph based on a natural language query. We frame SGM as a graph expansion task by introducing the incremental structure expanding (ISE) We construct a challenging dataset that contains more complicated queries and larger scene graphs than existing datasets.
  arXiv  Detail & Related papers  (2022-09-15T16:26:14Z)
• Fine-Grained Scene Graph Generation with Data Transfer [127.17675443137064]
  Scene graph generation (SGG) aims to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. We propose a novel Internal and External Data Transfer (IETrans) method, which can be applied in a play-and-plug fashion and expanded to large SGG with 1,807 predicate classes.
  arXiv  Detail & Related papers  (2022-03-22T12:26:56Z)
• Neural Graph Matching for Pre-training Graph Neural Networks [72.32801428070749]
  Graph neural networks (GNNs) have been shown powerful capacity at modeling structural data. We present a novel Graph Matching based GNN Pre-Training framework, called GMPT. The proposed method can be applied to fully self-supervised pre-training and coarse-grained supervised pre-training.
  arXiv  Detail & Related papers  (2022-03-03T09:53:53Z)
• Hyper-relationship Learning Network for Scene Graph Generation [95.6796681398668]
  We propose a hyper-relationship learning network, termed HLN, for scene graph generation. We evaluate HLN on the most popular SGG dataset, i.e., the Visual Genome dataset. For example, the proposed HLN improves the recall per relationship from 11.3% to 13.1%, and maintains the recall per image from 19.8% to 34.9%.
  arXiv  Detail & Related papers  (2022-02-15T09:26:16Z)
• Segmentation-grounded Scene Graph Generation [47.34166260639392]
  We propose a framework for pixel-level segmentation-grounded scene graph generation. Our framework is agnostic to the underlying scene graph generation method. It is learned in a multi-task manner with both target and auxiliary datasets.
  arXiv  Detail & Related papers  (2021-04-29T08:54:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.