Inferring and Leveraging Parts from Object Shape for Improving Semantic Image Synthesis

• URL: http://arxiv.org/abs/2305.19547v1
• Date: Wed, 31 May 2023 04:27:47 GMT
• Title: Inferring and Leveraging Parts from Object Shape for Improving Semantic Image Synthesis
• Authors: Yuxiang Wei, Zhilong Ji, Xiaohe Wu, Jinfeng Bai, Lei Zhang, Wangmeng Zuo
• Abstract summary: This paper presents to infer Parts from Object ShapE (iPOSE) and leverage it for improving semantic image synthesis. We learn a PartNet for predicting the object part map with the guidance of pre-defined support part maps. Experiments show that our iPOSE not only generates objects with rich part details, but also enables to control the image synthesis flexibly.
• Score: 64.05076727277431
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the progress in semantic image synthesis, it remains a challenging problem to generate photo-realistic parts from input semantic map. Integrating part segmentation map can undoubtedly benefit image synthesis, but is bothersome and inconvenient to be provided by users. To improve part synthesis, this paper presents to infer Parts from Object ShapE (iPOSE) and leverage it for improving semantic image synthesis. However, albeit several part segmentation datasets are available, part annotations are still not provided for many object categories in semantic image synthesis. To circumvent it, we resort to few-shot regime to learn a PartNet for predicting the object part map with the guidance of pre-defined support part maps. PartNet can be readily generalized to handle a new object category when a small number (e.g., 3) of support part maps for this category are provided. Furthermore, part semantic modulation is presented to incorporate both inferred part map and semantic map for image synthesis. Experiments show that our iPOSE not only generates objects with rich part details, but also enables to control the image synthesis flexibly. And our iPOSE performs favorably against the state-of-the-art methods in terms of quantitative and qualitative evaluation. Our code will be publicly available at https://github.com/csyxwei/iPOSE.

Related papers

• Semantic-shape Adaptive Feature Modulation for Semantic Image Synthesis [71.56830815617553]
  A fine-grained part-level semantic layout will benefit object details generation. A Shape-aware Position Descriptor (SPD) is proposed to describe each pixel's positional feature. A Semantic-shape Adaptive Feature Modulation (SAFM) block is proposed to combine the given semantic map and our positional features.
  arXiv  Detail & Related papers  (2022-03-31T09:06:04Z)
• PartImageNet: A Large, High-Quality Dataset of Parts [16.730418538593703]
  We propose PartImageNet, a high-quality dataset with part segmentation annotations. PartImageNet is unique because it offers part-level annotations on a general set of classes with non-rigid, articulated objects. It can be utilized in multiple vision tasks including but not limited to: Part Discovery, Few-shot Learning.
  arXiv  Detail & Related papers  (2021-12-02T02:12:03Z)
• 3D Compositional Zero-shot Learning with DeCompositional Consensus [102.7571947144639]
  We argue that part knowledge should be composable beyond the observed object classes. We present 3D Compositional Zero-shot Learning as a problem of part generalization from seen to unseen object classes.
  arXiv  Detail & Related papers  (2021-11-29T16:34:53Z)
• Generative View Synthesis: From Single-view Semantics to Novel-view Images [38.7873192939574]
  Generative View Synthesis (GVS) can synthesize multiple photorealistic views of a scene given a single semantic map. We first lift the input 2D semantic map onto a 3D layered representation of the scene in feature space. We then project the layered features onto the target views to generate the final novel-view images.
  arXiv  Detail & Related papers  (2020-08-20T17:48:16Z)
• Improving Semantic Segmentation via Decoupled Body and Edge Supervision [89.57847958016981]
  Existing semantic segmentation approaches either aim to improve the object's inner consistency by modeling the global context, or refine objects detail along their boundaries by multi-scale feature fusion. In this paper, a new paradigm for semantic segmentation is proposed. Our insight is that appealing performance of semantic segmentation requires textitexplicitly modeling the object textitbody and textitedge, which correspond to the high and low frequency of the image. We show that the proposed framework with various baselines or backbone networks leads to better object inner consistency and object boundaries.
  arXiv  Detail & Related papers  (2020-07-20T12:11:22Z)
• Controllable Image Synthesis via SegVAE [89.04391680233493]
  A semantic map is commonly used intermediate representation for conditional image generation. In this work, we specifically target at generating semantic maps given a label-set consisting of desired categories. The proposed framework, SegVAE, synthesizes semantic maps in an iterative manner using conditional variational autoencoder.
  arXiv  Detail & Related papers  (2020-07-16T15:18:53Z)
• Part-aware Prototype Network for Few-shot Semantic Segmentation [50.581647306020095]
  We propose a novel few-shot semantic segmentation framework based on the prototype representation. Our key idea is to decompose the holistic class representation into a set of part-aware prototypes. We develop a novel graph neural network model to generate and enhance the proposed part-aware prototypes.
  arXiv  Detail & Related papers  (2020-07-13T11:03:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.