Pluralistic Image Completion with Probabilistic Mixture-of-Experts

• URL: http://arxiv.org/abs/2205.09086v1
• Date: Wed, 18 May 2022 17:24:21 GMT
• Title: Pluralistic Image Completion with Probabilistic Mixture-of-Experts
• Authors: Xiaobo Xia, Wenhao Yang, Jie Ren, Yewen Li, Yibing Zhan, Bo Han, Tongliang Liu
• Abstract summary: We introduce a unified probabilistic graph model that represents the complex interactions in image completion. The entire procedure of image completion is then mathematically divided into several sub-procedures, which helps efficient enforcement of constraints. The inherent parameters of GMM are task-related, which are optimized adaptively during training, while the number of its primitives can control the diversity of results conveniently.
• Score: 58.81469985455467
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pluralistic image completion focuses on generating both visually realistic and diverse results for image completion. Prior methods enjoy the empirical successes of this task. However, their used constraints for pluralistic image completion are argued to be not well interpretable and unsatisfactory from two aspects. First, the constraints for visual reality can be weakly correlated to the objective of image completion or even redundant. Second, the constraints for diversity are designed to be task-agnostic, which causes the constraints to not work well. In this paper, to address the issues, we propose an end-to-end probabilistic method. Specifically, we introduce a unified probabilistic graph model that represents the complex interactions in image completion. The entire procedure of image completion is then mathematically divided into several sub-procedures, which helps efficient enforcement of constraints. The sub-procedure directly related to pluralistic results is identified, where the interaction is established by a Gaussian mixture model (GMM). The inherent parameters of GMM are task-related, which are optimized adaptively during training, while the number of its primitives can control the diversity of results conveniently. We formally establish the effectiveness of our method and demonstrate it with comprehensive experiments.

Related papers

• One Diffusion to Generate Them All [54.82732533013014]
  OneDiffusion is a versatile, large-scale diffusion model that supports bidirectional image synthesis and understanding. It enables conditional generation from inputs such as text, depth, pose, layout, and semantic maps. OneDiffusion allows for multi-view generation, camera pose estimation, and instant personalization using sequential image inputs.
  arXiv  Detail & Related papers  (2024-11-25T12:11:05Z)
• Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgetting [49.87694319431288]
  Generalist segmentation models are increasingly favored for diverse tasks involving various objects from different image sources. We propose a Comprehensive Generative (CGR) framework that restores appearance and semantic knowledge by synthesizing image-mask pairs. Experiments on incremental tasks (cardiac, fundus and prostate segmentation) show its clear advantage for alleviating concurrent appearance and semantic forgetting.
  arXiv  Detail & Related papers  (2024-06-28T10:05:58Z)
• SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow [94.90853153808987]
  Semantic segmentation and semantic image synthesis are representative tasks in visual perception and generation. We propose a unified framework (SemFlow) and model them as a pair of reverse problems. Experiments show that our SemFlow achieves competitive results on semantic segmentation and semantic image synthesis tasks.
  arXiv  Detail & Related papers  (2024-05-30T17:34:40Z)
• Multi-Task Learning with Multi-Task Optimization [31.518330903602095]
  We show that a set of optimized yet well-distributed models embody different trade-offs in one algorithmic pass. We investigate the proposed multi-task learning with multi-task optimization for solving various problem settings.
  arXiv  Detail & Related papers  (2024-03-24T14:04:40Z)
• Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Models [54.1843419649895]
  A major challenge in using generative models to accomplish this task is the lack of paired data containing all modalities and corresponding outputs. We propose a solution based on a denoising diffusion probabilistic synthesis models to generate images under multi-model priors.
  arXiv  Detail & Related papers  (2022-06-10T12:23:05Z)
• Large Scale Image Completion via Co-Modulated Generative Adversarial Networks [18.312552957727828]
  We propose a generic new approach that bridges the gap between image-conditional and recent unconditional generative architectures. Also, due to the lack of good quantitative metrics for image completion, we propose the new Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS) Experiments demonstrate superior performance in terms of both quality and diversity over state-of-the-art methods in free-form image completion and easy generalization to image-to-image translation.
  arXiv  Detail & Related papers  (2021-03-18T17:59:11Z)
• DiverseNet: When One Right Answer is not Enough [35.764028730120096]
  We introduce a simple method for training a neural network, which enables diverse structured predictions to be made for each test-time query. Our method results in quantitative improvements across three challenging tasks: 2D image completion, 3D volume estimation, and flow prediction.
  arXiv  Detail & Related papers  (2020-08-24T18:12:49Z)
• Grounded and Controllable Image Completion by Incorporating Lexical Semantics [111.47374576372813]
  Lexical Semantic Image Completion (LSIC) may have potential applications in art, design, and heritage conservation. We advocate generating results faithful to both visual and lexical semantic context. One major challenge for LSIC comes from modeling and aligning the structure of visual-semantic context.
  arXiv  Detail & Related papers  (2020-02-29T16:54:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.