Scene Graph to Image Generation with Contextualized Object Layout Refinement

• URL: http://arxiv.org/abs/2009.10939v4
• Date: Mon, 10 Oct 2022 20:33:51 GMT
• Title: Scene Graph to Image Generation with Contextualized Object Layout Refinement
• Authors: Maor Ivgi, Yaniv Benny, Avichai Ben-David, Jonathan Berant, and Lior Wolf
• Abstract summary: We propose a novel method to generate images from scene graphs. Our approach improves the layout coverage by almost 20 points and drops object overlap to negligible amounts.
• Score: 92.85331019618332
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating images from scene graphs is a challenging task that attracted substantial interest recently. Prior works have approached this task by generating an intermediate layout description of the target image. However, the representation of each object in the layout was generated independently, which resulted in high overlap, low coverage, and an overall blurry layout. We propose a novel method that alleviates these issues by generating the entire layout description gradually to improve inter-object dependency. We empirically show on the COCO-STUFF dataset that our approach improves the quality of both the intermediate layout and the final image. Our approach improves the layout coverage by almost 20 points and drops object overlap to negligible amounts.

Related papers

• LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation [121.45667242282721]
  We propose a coarse-to-fine paradigm to achieve layout planning and image generation. Our proposed method outperforms the state-of-the-art models in terms of photorealistic layout and image generation.
  arXiv  Detail & Related papers  (2023-08-09T17:45:04Z)
• Geometry Aligned Variational Transformer for Image-conditioned Layout Generation [38.747175229902396]
  We propose an Image-Conditioned Variational Transformer (ICVT) that autoregressively generates various layouts in an image. First, self-attention mechanism is adopted to model the contextual relationship within layout elements, while cross-attention mechanism is used to fuse the visual information of conditional images. We construct a large-scale advertisement poster layout designing dataset with delicate layout and saliency map annotations.
  arXiv  Detail & Related papers  (2022-09-02T07:19:12Z)
• Iterative Scene Graph Generation [55.893695946885174]
  Scene graph generation involves identifying object entities and their corresponding interaction predicates in a given image (or video) Existing approaches to scene graph generation assume certain factorization of the joint distribution to make the estimation iteration feasible. We propose a novel framework that addresses this limitation, as well as introduces dynamic conditioning on the image.
  arXiv  Detail & Related papers  (2022-07-27T10:37:29Z)
• Scenes and Surroundings: Scene Graph Generation using Relation Transformer [13.146732454123326]
  This work proposes a novel local-context aware architecture named relation transformer. Our hierarchical multi-head attention-based approach efficiently captures contextual dependencies between objects and predicts their relationships. In comparison to state-of-the-art approaches, we have achieved an overall mean textbf4.85% improvement.
  arXiv  Detail & Related papers  (2021-07-12T14:22:20Z)
• Segmentation-grounded Scene Graph Generation [47.34166260639392]
  We propose a framework for pixel-level segmentation-grounded scene graph generation. Our framework is agnostic to the underlying scene graph generation method. It is learned in a multi-task manner with both target and auxiliary datasets.
  arXiv  Detail & Related papers  (2021-04-29T08:54:08Z)
• Semantic Layout Manipulation with High-Resolution Sparse Attention [106.59650698907953]
  We tackle the problem of semantic image layout manipulation, which aims to manipulate an input image by editing its semantic label map. A core problem of this task is how to transfer visual details from the input images to the new semantic layout while making the resulting image visually realistic. We propose a high-resolution sparse attention module that effectively transfers visual details to new layouts at a resolution up to 512x512.
  arXiv  Detail & Related papers  (2020-12-14T06:50:43Z)
• Perspective Plane Program Induction from a Single Image [85.28956922100305]
  We study the inverse graphics problem of inferring a holistic representation for natural images. We formulate this problem as jointly finding the camera pose and scene structure that best describe the input image. Our proposed framework, Perspective Plane Program Induction (P3I), combines search-based and gradient-based algorithms to efficiently solve the problem.
  arXiv  Detail & Related papers  (2020-06-25T21:18:58Z)
• Object-Centric Image Generation from Layouts [93.10217725729468]
  We develop a layout-to-image-generation method to generate complex scenes with multiple objects. Our method learns representations of the spatial relationships between objects in the scene, which lead to our model's improved layout-fidelity. We introduce SceneFID, an object-centric adaptation of the popular Fr'echet Inception Distance metric, that is better suited for multi-object images.
  arXiv  Detail & Related papers  (2020-03-16T21:40:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.