I$^2$SB: Image-to-Image Schr\"odinger Bridge
- URL: http://arxiv.org/abs/2302.05872v3
- Date: Fri, 26 May 2023 02:55:08 GMT
- Title: I$^2$SB: Image-to-Image Schr\"odinger Bridge
- Authors: Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A. Theodorou,
Weili Nie, Anima Anandkumar
- Abstract summary: Image-to-Image Schr"odinger Bridge (I$2$SB) is a new class of conditional diffusion models.
I$2$SB directly learns the nonlinear diffusion processes between two given distributions.
We show that I$2$SB surpasses standard conditional diffusion models with more interpretable generative processes.
- Score: 87.43524087956457
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose Image-to-Image Schr\"odinger Bridge (I$^2$SB), a new class of
conditional diffusion models that directly learn the nonlinear diffusion
processes between two given distributions. These diffusion bridges are
particularly useful for image restoration, as the degraded images are
structurally informative priors for reconstructing the clean images. I$^2$SB
belongs to a tractable class of Schr\"odinger bridge, the nonlinear extension
to score-based models, whose marginal distributions can be computed
analytically given boundary pairs. This results in a simulation-free framework
for nonlinear diffusions, where the I$^2$SB training becomes scalable by
adopting practical techniques used in standard diffusion models. We validate
I$^2$SB in solving various image restoration tasks, including inpainting,
super-resolution, deblurring, and JPEG restoration on ImageNet 256x256 and show
that I$^2$SB surpasses standard conditional diffusion models with more
interpretable generative processes. Moreover, I$^2$SB matches the performance
of inverse methods that additionally require the knowledge of the corruption
operators. Our work opens up new algorithmic opportunities for developing
efficient nonlinear diffusion models on a large scale. scale. Project page and
codes: https://i2sb.github.io/
Related papers
- $\infty$-Brush: Controllable Large Image Synthesis with Diffusion Models in Infinite Dimensions [58.42011190989414]
We introduce a novel conditional diffusion model in infinite dimensions, $infty$-Brush for controllable large image synthesis.
To our best knowledge, $infty$-Brush is the first conditional diffusion model in function space, that can controllably synthesize images at arbitrary resolutions of up to $4096times4096$ pixels.
arXiv Detail & Related papers (2024-07-20T00:04:49Z) - FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models [56.71672127740099]
We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets.
We leverage different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation.
Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets.
arXiv Detail & Related papers (2024-03-29T10:38:25Z) - Cross-view Masked Diffusion Transformers for Person Image Synthesis [21.242398582282522]
We present X-MDPT, a novel diffusion model designed for pose-guided human image generation.
X-MDPT distinguishes itself by employing masked diffusion transformers that operate on latent patches.
Our model outperforms state-of-the-art approaches on the DeepFashion dataset.
arXiv Detail & Related papers (2024-02-02T15:57:13Z) - Denoising Diffusion Bridge Models [54.87947768074036]
Diffusion models are powerful generative models that map noise to data using processes.
For many applications such as image editing, the model input comes from a distribution that is not random noise.
In our work, we propose Denoising Diffusion Bridge Models (DDBMs)
arXiv Detail & Related papers (2023-09-29T03:24:24Z) - Improving Generative Model-based Unfolding with Schr\"{o}dinger Bridges [14.989614554242229]
Machine learning-based unfolding has enabled unbinned and high-dimensional differential cross section measurements.
We propose to use Schroedinger Bridges and diffusion models to create SBUnfold, an unfolding approach that combines the strengths of both discriminative and generative models.
We show that SBUnfold achieves excellent performance compared to state of the art methods on a synthetic Z+jets dataset.
arXiv Detail & Related papers (2023-08-23T18:01:01Z) - Unpaired Image-to-Image Translation via Neural Schr\"odinger Bridge [70.79973551604539]
We propose Unpaired Neural Schr"odinger Bridge (UNSB), which expresses the SB problem as a sequence of adversarial learning problems.
UNSB is scalable and successfully solves various unpaired I2I translation tasks.
arXiv Detail & Related papers (2023-05-24T12:05:24Z) - SDM: Spatial Diffusion Model for Large Hole Image Inpainting [106.90795513361498]
We present a novel spatial diffusion model (SDM) that uses a few iterations to gradually deliver informative pixels to the entire image.
Also, thanks to the proposed decoupled probabilistic modeling and spatial diffusion scheme, our method achieves high-quality large-hole completion.
arXiv Detail & Related papers (2022-12-06T13:30:18Z) - Bessel Equivariant Networks for Inversion of Transmission Effects in
Multi-Mode Optical Fibres [3.2981146586835703]
We develop a new type of model for solving the task of inverting the transmission effects of multi-mode optical fibres.
We use the azimuthal correlations known to exist in fibre speckle patterns to account for the difference in spatial arrangement between input and speckle patterns.
This model can scale to previously unachievable resolutions of imaging with multi-mode optical fibres and is demonstrated on $256 times 256$ pixel images.
arXiv Detail & Related papers (2022-07-26T12:29:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.