Deep Optimal Transport: A Practical Algorithm for Photo-realistic Image
Restoration
- URL: http://arxiv.org/abs/2306.02342v1
- Date: Sun, 4 Jun 2023 12:21:53 GMT
- Title: Deep Optimal Transport: A Practical Algorithm for Photo-realistic Image
Restoration
- Authors: Theo Adrai, Guy Ohayon, Tomer Michaeli and Michael Elad
- Abstract summary: We propose an image restoration algorithm that can control the perceptual quality and/or the mean square error (MSE) of any pre-trained model.
Given about a dozen images restored by the model, it can significantly improve the perceptual quality and/or the MSE of the model for newly restored images without further training.
- Score: 44.47246905244631
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We propose an image restoration algorithm that can control the perceptual
quality and/or the mean square error (MSE) of any pre-trained model, trading
one over the other at test time. Our algorithm is few-shot: Given about a dozen
images restored by the model, it can significantly improve the perceptual
quality and/or the MSE of the model for newly restored images without further
training. Our approach is motivated by a recent theoretical result that links
between the minimum MSE (MMSE) predictor and the predictor that minimizes the
MSE under a perfect perceptual quality constraint. Specifically, it has been
shown that the latter can be obtained by optimally transporting the output of
the former, such that its distribution matches the source data. Thus, to
improve the perceptual quality of a predictor that was originally trained to
minimize MSE, we approximate the optimal transport by a linear transformation
in the latent space of a variational auto-encoder, which we compute in
closed-form using empirical means and covariances. Going beyond the theory, we
find that applying the same procedure on models that were initially trained to
achieve high perceptual quality, typically improves their perceptual quality
even further. And by interpolating the results with the original output of the
model, we can improve their MSE on the expense of perceptual quality. We
illustrate our method on a variety of degradations applied to general content
images of arbitrary dimensions.
Related papers
- Exploiting Diffusion Prior for Real-World Image Super-Resolution [75.5898357277047]
We present a novel approach to leverage prior knowledge encapsulated in pre-trained text-to-image diffusion models for blind super-resolution.
By employing our time-aware encoder, we can achieve promising restoration results without altering the pre-trained synthesis model.
arXiv Detail & Related papers (2023-05-11T17:55:25Z) - Perception-Distortion Balanced ADMM Optimization for Single-Image
Super-Resolution [29.19388490351459]
We propose a novel super-resolution model with a low-frequency constraint (LFc-SR)
We introduce an ADMM-based alternating optimization method for the non-trivial learning of the constrained model.
Experiments showed that our method, without cumbersome post-processing procedures, achieved the state-of-the-art performance.
arXiv Detail & Related papers (2022-08-05T05:37:55Z) - PatchNR: Learning from Small Data by Patch Normalizing Flow
Regularization [57.37911115888587]
We introduce a regularizer for the variational modeling of inverse problems in imaging based on normalizing flows.
Our regularizer, called patchNR, involves a normalizing flow learned on patches of very few images.
arXiv Detail & Related papers (2022-05-24T12:14:26Z) - Paired Image-to-Image Translation Quality Assessment Using Multi-Method
Fusion [0.0]
This paper proposes a novel approach that combines signals of image quality between paired source and transformation to predict the latter's similarity with a hypothetical ground truth.
We trained a Multi-Method Fusion (MMF) model via an ensemble of gradient-boosted regressors to predict Deep Image Structure and Texture Similarity (DISTS)
Analysis revealed the task to be feature-constrained, introducing a trade-off at inference between metric time and prediction accuracy.
arXiv Detail & Related papers (2022-05-09T11:05:15Z) - Stable Optimization for Large Vision Model Based Deep Image Prior in
Cone-Beam CT Reconstruction [6.558735319783205]
Large Vision Model (LVM) has recently demonstrated great potential for medical imaging tasks.
Deep Image Prior (DIP) effectively guides an untrained neural network to generate high-quality CBCT images without any training data.
We propose a stable optimization method for the forward-model-free DIP model for sparse-view CBCT.
arXiv Detail & Related papers (2022-03-23T15:16:29Z) - Image-specific Convolutional Kernel Modulation for Single Image
Super-resolution [85.09413241502209]
In this issue, we propose a novel image-specific convolutional modulation kernel (IKM)
We exploit the global contextual information of image or feature to generate an attention weight for adaptively modulating the convolutional kernels.
Experiments on single image super-resolution show that the proposed methods achieve superior performances over state-of-the-art methods.
arXiv Detail & Related papers (2021-11-16T11:05:10Z) - Perceptual Image Restoration with High-Quality Priori and Degradation
Learning [28.93489249639681]
We show that our model performs well in measuring the similarity between restored and degraded images.
Our simultaneous restoration and enhancement framework generalizes well to real-world complicated degradation types.
arXiv Detail & Related papers (2021-03-04T13:19:50Z) - Deep Variational Network Toward Blind Image Restoration [60.45350399661175]
Blind image restoration is a common yet challenging problem in computer vision.
We propose a novel blind image restoration method, aiming to integrate both the advantages of them.
Experiments on two typical blind IR tasks, namely image denoising and super-resolution, demonstrate that the proposed method achieves superior performance over current state-of-the-arts.
arXiv Detail & Related papers (2020-08-25T03:30:53Z) - Perceptually Optimizing Deep Image Compression [53.705543593594285]
Mean squared error (MSE) and $ell_p$ norms have largely dominated the measurement of loss in neural networks.
We propose a different proxy approach to optimize image analysis networks against quantitative perceptual models.
arXiv Detail & Related papers (2020-07-03T14:33:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.