MIGS: Meta Image Generation from Scene Graphs

• URL: http://arxiv.org/abs/2110.11918v1
• Date: Fri, 22 Oct 2021 17:02:44 GMT
• Title: MIGS: Meta Image Generation from Scene Graphs
• Authors: Azade Farshad, Sabrina Musatian, Helisa Dhamo, Nassir Navab
• Abstract summary: We propose MIGS (Meta Image Generation from Scene Graphs), a meta-learning based approach for few-shot image generation from graphs. By sampling the data in a task-driven fashion, we train the generator using meta-learning on different sets of tasks that are categorized based on the scene attributes. Our results show that using this meta-learning approach for the generation of images from scene graphs state-of-the-art performance in terms of image quality and capturing the semantic relationships in the scene.
• Score: 48.82382997154196
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generation of images from scene graphs is a promising direction towards explicit scene generation and manipulation. However, the images generated from the scene graphs lack quality, which in part comes due to high difficulty and diversity in the data. We propose MIGS (Meta Image Generation from Scene Graphs), a meta-learning based approach for few-shot image generation from graphs that enables adapting the model to different scenes and increases the image quality by training on diverse sets of tasks. By sampling the data in a task-driven fashion, we train the generator using meta-learning on different sets of tasks that are categorized based on the scene attributes. Our results show that using this meta-learning approach for the generation of images from scene graphs achieves state-of-the-art performance in terms of image quality and capturing the semantic relationships in the scene. Project Website: https://migs2021.github.io/

Related papers

• Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation [44.457347230146404]
  We leverage the scene graph, a powerful structured representation, for complex image generation. We employ the generative capabilities of variational autoencoders and diffusion models in a generalizable manner. Our method outperforms recent competitors based on text, layout, or scene graph.
  arXiv  Detail & Related papers  (2024-10-01T07:02:46Z)
• FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing [66.70054075041487]
  Existing scene graphs that convert image captions into scene graphs often suffer from two types of errors. First, the generated scene graphs fail to capture the true semantics of the captions or the corresponding images, resulting in a lack of faithfulness. Second, the generated scene graphs have high inconsistency, with the same semantics represented by different annotations.
  arXiv  Detail & Related papers  (2023-05-27T15:38:31Z)
• SPAN: Learning Similarity between Scene Graphs and Images with Transformers [29.582313604112336]
  We propose a Scene graPh-imAge coNtrastive learning framework, SPAN, that can measure the similarity between scene graphs and images. We introduce a novel graph serialization technique that transforms a scene graph into a sequence with structural encodings.
  arXiv  Detail & Related papers  (2023-04-02T18:13:36Z)
• Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training [112.94542676251133]
  We propose to learn scene graph embeddings by directly optimizing their alignment with images. Specifically, we pre-train an encoder to extract both global and local information from scene graphs. The resulting method, called SGDiff, allows for the semantic manipulation of generated images by modifying scene graph nodes and connections.
  arXiv  Detail & Related papers  (2022-11-21T01:11:19Z)
• Learning to Generate Scene Graph from Natural Language Supervision [52.18175340725455]
  We propose one of the first methods that learn from image-sentence pairs to extract a graphical representation of localized objects and their relationships within an image, known as scene graph. We leverage an off-the-shelf object detector to identify and localize object instances, match labels of detected regions to concepts parsed from captions, and thus create "pseudo" labels for learning scene graph.
  arXiv  Detail & Related papers  (2021-09-06T03:38:52Z)
• Unconditional Scene Graph Generation [72.53624470737712]
  We develop a deep auto-regressive model called SceneGraphGen which can learn the probability distribution over labelled and directed graphs. We show that the scene graphs generated by SceneGraphGen are diverse and follow the semantic patterns of real-world scenes.
  arXiv  Detail & Related papers  (2021-08-12T17:57:16Z)
• Image Scene Graph Generation (SGG) Benchmark [58.33119409657256]
  There is a surge of interest in image scene graph generation (object, and relationship detection) Due to the lack of a good benchmark, the reported results of different scene graph generation models are not directly comparable. We have developed a much-needed scene graph generation benchmark based on the maskrcnn-benchmark and several popular models.
  arXiv  Detail & Related papers  (2021-07-27T05:10:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.