Deep Optimal Transport: A Practical Algorithm for Photo-realistic Image Restoration

• URL: http://arxiv.org/abs/2306.02342v2
• Date: Mon, 12 Aug 2024 08:34:12 GMT
• Title: Deep Optimal Transport: A Practical Algorithm for Photo-realistic Image Restoration
• Authors: Theo Adrai, Guy Ohayon, Tomer Michaeli, Michael Elad,
• Abstract summary: We propose an image restoration algorithm that can control the perceptual quality and/or the mean square error (MSE) of any pre-trained model. Given about a dozen images restored by the model, it can significantly improve the perceptual quality and/or the MSE of the model for newly restored images without further training.
• Score: 31.58365182858562
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose an image restoration algorithm that can control the perceptual quality and/or the mean square error (MSE) of any pre-trained model, trading one over the other at test time. Our algorithm is few-shot: Given about a dozen images restored by the model, it can significantly improve the perceptual quality and/or the MSE of the model for newly restored images without further training. Our approach is motivated by a recent theoretical result that links between the minimum MSE (MMSE) predictor and the predictor that minimizes the MSE under a perfect perceptual quality constraint. Specifically, it has been shown that the latter can be obtained by optimally transporting the output of the former, such that its distribution matches the source data. Thus, to improve the perceptual quality of a predictor that was originally trained to minimize MSE, we approximate the optimal transport by a linear transformation in the latent space of a variational auto-encoder, which we compute in closed-form using empirical means and covariances. Going beyond the theory, we find that applying the same procedure on models that were initially trained to achieve high perceptual quality, typically improves their perceptual quality even further. And by interpolating the results with the original output of the model, we can improve their MSE on the expense of perceptual quality. We illustrate our method on a variety of degradations applied to general content images of arbitrary dimensions.

Related papers

• Posterior-Mean Rectified Flow: Towards Minimum MSE Photo-Realistic Image Restoration [34.50287066865267]
  Posterior-Mean Rectified Flow (PMRF) is a simple yet highly effective algorithm that approximates this optimal estimator. We investigate the theoretical utility of PMRF and demonstrate that it consistently outperforms previous methods on a variety of image restoration tasks.
  arXiv  Detail & Related papers  (2024-10-01T05:54:07Z)
• Exploiting Diffusion Prior for Real-World Image Super-Resolution [75.5898357277047]
  We present a novel approach to leverage prior knowledge encapsulated in pre-trained text-to-image diffusion models for blind super-resolution. By employing our time-aware encoder, we can achieve promising restoration results without altering the pre-trained synthesis model.
  arXiv  Detail & Related papers  (2023-05-11T17:55:25Z)
• Perception-Distortion Balanced ADMM Optimization for Single-Image Super-Resolution [29.19388490351459]
  We propose a novel super-resolution model with a low-frequency constraint (LFc-SR) We introduce an ADMM-based alternating optimization method for the non-trivial learning of the constrained model. Experiments showed that our method, without cumbersome post-processing procedures, achieved the state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-08-05T05:37:55Z)
• PatchNR: Learning from Small Data by Patch Normalizing Flow Regularization [57.37911115888587]
  We introduce a regularizer for the variational modeling of inverse problems in imaging based on normalizing flows. Our regularizer, called patchNR, involves a normalizing flow learned on patches of very few images.
  arXiv  Detail & Related papers  (2022-05-24T12:14:26Z)
• Stable Optimization for Large Vision Model Based Deep Image Prior in Cone-Beam CT Reconstruction [6.558735319783205]
  Large Vision Model (LVM) has recently demonstrated great potential for medical imaging tasks. Deep Image Prior (DIP) effectively guides an untrained neural network to generate high-quality CBCT images without any training data. We propose a stable optimization method for the forward-model-free DIP model for sparse-view CBCT.
  arXiv  Detail & Related papers  (2022-03-23T15:16:29Z)
• Image-specific Convolutional Kernel Modulation for Single Image Super-resolution [85.09413241502209]
  In this issue, we propose a novel image-specific convolutional modulation kernel (IKM) We exploit the global contextual information of image or feature to generate an attention weight for adaptively modulating the convolutional kernels. Experiments on single image super-resolution show that the proposed methods achieve superior performances over state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-16T11:05:10Z)
• Perceptual Image Restoration with High-Quality Priori and Degradation Learning [28.93489249639681]
  We show that our model performs well in measuring the similarity between restored and degraded images. Our simultaneous restoration and enhancement framework generalizes well to real-world complicated degradation types.
  arXiv  Detail & Related papers  (2021-03-04T13:19:50Z)
• Deep Variational Network Toward Blind Image Restoration [60.45350399661175]
  Blind image restoration is a common yet challenging problem in computer vision. We propose a novel blind image restoration method, aiming to integrate both the advantages of them. Experiments on two typical blind IR tasks, namely image denoising and super-resolution, demonstrate that the proposed method achieves superior performance over current state-of-the-arts.
  arXiv  Detail & Related papers  (2020-08-25T03:30:53Z)
• Perceptually Optimizing Deep Image Compression [53.705543593594285]
  Mean squared error (MSE) and $ell_p$ norms have largely dominated the measurement of loss in neural networks. We propose a different proxy approach to optimize image analysis networks against quantitative perceptual models.
  arXiv  Detail & Related papers  (2020-07-03T14:33:28Z)
• Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild [98.48284827503409]
  We develop a textitunified BIQA model and an approach of training it for both synthetic and realistic distortions. We employ the fidelity loss to optimize a deep neural network for BIQA over a large number of such image pairs. Experiments on six IQA databases show the promise of the learned method in blindly assessing image quality in the laboratory and wild.
  arXiv  Detail & Related papers  (2020-05-28T13:35:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.