Test-Time Adaptation for Super-Resolution: You Only Need to Overfit on a Few More Images

• URL: http://arxiv.org/abs/2104.02663v1
• Date: Tue, 6 Apr 2021 16:50:52 GMT
• Title: Test-Time Adaptation for Super-Resolution: You Only Need to Overfit on a Few More Images
• Authors: Mohammad Saeed Rad, Thomas Yu, Behzad Bozorgtabar, Jean-Philippe Thiran
• Abstract summary: We propose a simple yet universal approach to improve the perceptual quality of the HR prediction from a pre-trained SR network. We show the effects of fine-tuning on images in terms of the perceptual quality and PSNR/SSIM values.
• Score: 12.846479438896338
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing reference (RF)-based super-resolution (SR) models try to improve perceptual quality in SR under the assumption of the availability of high-resolution RF images paired with low-resolution (LR) inputs at testing. As the RF images should be similar in terms of content, colors, contrast, etc. to the test image, this hinders the applicability in a real scenario. Other approaches to increase the perceptual quality of images, including perceptual loss and adversarial losses, tend to dramatically decrease fidelity to the ground-truth through significant decreases in PSNR/SSIM. Addressing both issues, we propose a simple yet universal approach to improve the perceptual quality of the HR prediction from a pre-trained SR network on a given LR input by further fine-tuning the SR network on a subset of images from the training dataset with similar patterns of activation as the initial HR prediction, with respect to the filters of a feature extractor. In particular, we show the effects of fine-tuning on these images in terms of the perceptual quality and PSNR/SSIM values. Contrary to perceptually driven approaches, we demonstrate that the fine-tuned network produces a HR prediction with both greater perceptual quality and minimal changes to the PSNR/SSIM with respect to the initial HR prediction. Further, we present novel numerical experiments concerning the filters of SR networks, where we show through filter correlation, that the filters of the fine-tuned network from our method are closer to "ideal" filters, than those of the baseline network or a network fine-tuned on random images.

Related papers

• HF-Diff: High-Frequency Perceptual Loss and Distribution Matching for One-Step Diffusion-Based Image Super-Resolution [0.0]
  We introduce a high-frequency perceptual loss by utilizing an invertible neural network (INN) pretrained on the ImageNet dataset. During the training phase, we impose to preserve the high-frequency features of super-resolved and ground truth (GT) images. We achieve a state-of-the-art CLIPIQA score in the benchmark RealSR, RealSet65, DIV2K-Val, and ImageNet datasets.
  arXiv  Detail & Related papers  (2024-11-20T18:56:24Z)
• Hierarchical Similarity Learning for Aliasing Suppression Image Super-Resolution [64.15915577164894]
  A hierarchical image super-resolution network (HSRNet) is proposed to suppress the influence of aliasing. HSRNet achieves better quantitative and visual performance than other works, and remits the aliasing more effectively.
  arXiv  Detail & Related papers  (2022-06-07T14:55:32Z)
• Frequency-Aware Physics-Inspired Degradation Model for Real-World Image Super-Resolution [18.328806055594576]
  We formulate a real-world physics-inspired degradationmodel by considering bothoptics andsensordegradation. We propose to use a convolutional neural network (CNN) to learn the cutoff frequency of real-world degradation process. We evaluatethe effectiveness and generalization capability of the proposed degradation model on real-world images captured by different imaging systems.
  arXiv  Detail & Related papers  (2021-11-05T07:30:00Z)
• Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling [139.25215100378284]
  We propose a hierarchical conditional flow (HCFlow) as a unified framework for image SR and image rescaling. HCFlow learns a mapping between HR and LR image pairs by modelling the distribution of the LR image and the rest high-frequency component simultaneously. To further enhance the performance, other losses such as perceptual loss and GAN loss are combined with the commonly used negative log-likelihood loss in training.
  arXiv  Detail & Related papers  (2021-08-11T16:11:01Z)
• Perception Consistency Ultrasound Image Super-resolution via Self-supervised CycleGAN [63.49373689654419]
  We propose a new perception consistency ultrasound image super-resolution (SR) method based on self-supervision and cycle generative adversarial network (CycleGAN) We first generate the HR fathers and the LR sons of the test ultrasound LR image through image enhancement. We then make full use of the cycle loss of LR-SR-LR and HR-LR-SR and the adversarial characteristics of the discriminator to promote the generator to produce better perceptually consistent SR results.
  arXiv  Detail & Related papers  (2020-12-28T08:24:04Z)
• Frequency Consistent Adaptation for Real World Super Resolution [64.91914552787668]
  We propose a novel Frequency Consistent Adaptation (FCA) that ensures the frequency domain consistency when applying Super-Resolution (SR) methods to the real scene. We estimate degradation kernels from unsupervised images and generate the corresponding Low-Resolution (LR) images. Based on the domain-consistent LR-HR pairs, we train easy-implemented Convolutional Neural Network (CNN) SR models.
  arXiv  Detail & Related papers  (2020-12-18T08:25:39Z)
• Perceptually Optimizing Deep Image Compression [53.705543593594285]
  Mean squared error (MSE) and $ell_p$ norms have largely dominated the measurement of loss in neural networks. We propose a different proxy approach to optimize image analysis networks against quantitative perceptual models.
  arXiv  Detail & Related papers  (2020-07-03T14:33:28Z)
• Iterative Network for Image Super-Resolution [69.07361550998318]
  Single image super-resolution (SISR) has been greatly revitalized by the recent development of convolutional neural networks (CNN) This paper provides a new insight on conventional SISR algorithm, and proposes a substantially different approach relying on the iterative optimization. A novel iterative super-resolution network (ISRN) is proposed on top of the iterative optimization.
  arXiv  Detail & Related papers  (2020-05-20T11:11:47Z)
• Unpaired Image Super-Resolution using Pseudo-Supervision [12.18340575383456]
  We propose an unpaired image super-resolution (SR) method using a generative adversarial network. Our network consists of an unpaired kernel/noise correction network and a pseudo-paired SR network. Experiments on diverse datasets show that the proposed method is superior to existing solutions to the unpaired SR problem.
  arXiv  Detail & Related papers  (2020-02-26T10:30:52Z)
• Fast Adaptation to Super-Resolution Networks via Meta-Learning [24.637337634643885]
  In this work, we observe the opportunity for further improvement of the performance of SISR without changing the architecture of conventional SR networks. In the training stage, we train the network via meta-learning; thus, the network can quickly adapt to any input image at test time. We demonstrate that the proposed model-agnostic approach consistently improves the performance of conventional SR networks on various benchmark SR datasets.
  arXiv  Detail & Related papers  (2020-01-09T09:59:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.