HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation

• URL: http://arxiv.org/abs/2403.12033v1
• Date: Mon, 18 Mar 2024 17:59:10 GMT
• Title: HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation
• Authors: Ce Zhang, Simon Stepputtis, Joseph Campbell, Katia Sycara, Yaqi Xie,
• Abstract summary: A common approach enabling the ability to reason over visual data is Scene Graph Generation (SGG) We propose a novel SGG benchmark containing procedurally generated weather corruptions and other transformations over the Visual Genome dataset. We show that HiKER-SGG does not only demonstrate superior performance on corrupted images in a zero-shot manner, but also outperforms current state-of-the-art methods on uncorrupted SGG tasks.
• Score: 13.929906773382752
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Being able to understand visual scenes is a precursor for many downstream tasks, including autonomous driving, robotics, and other vision-based approaches. A common approach enabling the ability to reason over visual data is Scene Graph Generation (SGG); however, many existing approaches assume undisturbed vision, i.e., the absence of real-world corruptions such as fog, snow, smoke, as well as non-uniform perturbations like sun glare or water drops. In this work, we propose a novel SGG benchmark containing procedurally generated weather corruptions and other transformations over the Visual Genome dataset. Further, we introduce a corresponding approach, Hierarchical Knowledge Enhanced Robust Scene Graph Generation (HiKER-SGG), providing a strong baseline for scene graph generation under such challenging setting. At its core, HiKER-SGG utilizes a hierarchical knowledge graph in order to refine its predictions from coarse initial estimates to detailed predictions. In our extensive experiments, we show that HiKER-SGG does not only demonstrate superior performance on corrupted images in a zero-shot manner, but also outperforms current state-of-the-art methods on uncorrupted SGG tasks. Code is available at https://github.com/zhangce01/HiKER-SGG.

Related papers

• Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention [69.36723767339001]
  Scene Graph Generation (SGG) offers a structured representation critical in many computer vision applications. We propose a unified framework named OvSGTR towards fully open vocabulary SGG from a holistic view. For the more challenging settings of relation-involved open vocabulary SGG, the proposed approach integrates relation-aware pretraining.
  arXiv  Detail & Related papers  (2023-11-18T06:49:17Z)
• Fine-Grained is Too Coarse: A Novel Data-Centric Approach for Efficient Scene Graph Generation [0.7851536646859476]
  We introduce the task of Efficient Scene Graph Generation (SGG) that prioritizes the generation of relevant relations. We present a new dataset, VG150-curated, based on the annotations of the popular Visual Genome dataset. We show through a set of experiments that this dataset contains more high-quality and diverse annotations than the one usually use in SGG.
  arXiv  Detail & Related papers  (2023-05-30T00:55:49Z)
• Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World [67.03968403301143]
  Scene Graph Generation (SGG) aims to extract subject, predicate, object> relationships in images for vision understanding. Existing re-balancing strategies try to handle it via prior rules but are still confined to pre-defined conditions. We propose a Cross-modal prediCate boosting (CaCao) framework, where a visually-prompted language model is learned to generate diverse fine-grained predicates.
  arXiv  Detail & Related papers  (2023-03-23T13:06:38Z)
• Iterative Scene Graph Generation with Generative Transformers [6.243995448840211]
  Scene graphs provide a rich, structured representation of a scene by encoding the entities (objects) and their spatial relationships in a graphical format. Current approaches take a generation-by-classification approach where the scene graph is generated through labeling of all possible edges between objects in a scene. This work introduces a generative transformer-based approach to generating scene graphs beyond link prediction.
  arXiv  Detail & Related papers  (2022-11-30T00:05:44Z)
• Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning [84.39787427288525]
  Scene graph generation (SGG) is a fundamental task aimed at detecting visual relations between objects in an image. We introduce open-vocabulary scene graph generation, a novel, realistic and challenging setting in which a model is trained on a set of base object classes. Our method can support inference over completely unseen object classes, which existing methods are incapable of handling.
  arXiv  Detail & Related papers  (2022-08-17T09:05:38Z)
• Learning To Generate Scene Graph from Head to Tail [65.48134724633472]
  We propose a novel SGG framework, learning to generate scene graphs from Head to Tail (SGG-HT) CRM learns head/easy samples firstly for robust features of head predicates and then gradually focuses on tail/hard ones. SCM is proposed to relieve semantic deviation by ensuring the semantic consistency between the generated scene graph and the ground truth in global and local representations.
  arXiv  Detail & Related papers  (2022-06-23T12:16:44Z)
• Fine-Grained Scene Graph Generation with Data Transfer [127.17675443137064]
  Scene graph generation (SGG) aims to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. We propose a novel Internal and External Data Transfer (IETrans) method, which can be applied in a play-and-plug fashion and expanded to large SGG with 1,807 predicate classes.
  arXiv  Detail & Related papers  (2022-03-22T12:26:56Z)
• Unbiased Scene Graph Generation from Biased Training [99.88125954889937]
  We present a novel SGG framework based on causal inference but not the conventional likelihood. We propose to draw the counterfactual causality from the trained graph to infer the effect from the bad bias. In particular, we use Total Direct Effect (TDE) as the proposed final predicate score for unbiased SGG.
  arXiv  Detail & Related papers  (2020-02-27T07:29:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.