Real-time 6K Image Rescaling with Rate-distortion Optimization

• URL: http://arxiv.org/abs/2304.01064v2
• Date: Fri, 19 May 2023 12:34:17 GMT
• Title: Real-time 6K Image Rescaling with Rate-distortion Optimization
• Authors: Chenyang Qi, Xin Yang, Ka Leong Cheng, Ying-Cong Chen, Qifeng Chen
• Abstract summary: We propose a novel framework for real-time 6K rate-distortion-aware image rescaling. Our framework embeds an HR image into a JPEG LR thumbnail by an encoder. Then, an efficient frequency-aware decoder reconstructs a high-fidelity HR image from the LR one in real time.
• Score: 53.61374658914277
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contemporary image rescaling aims at embedding a high-resolution (HR) image into a low-resolution (LR) thumbnail image that contains embedded information for HR image reconstruction. Unlike traditional image super-resolution, this enables high-fidelity HR image restoration faithful to the original one, given the embedded information in the LR thumbnail. However, state-of-the-art image rescaling methods do not optimize the LR image file size for efficient sharing and fall short of real-time performance for ultra-high-resolution (e.g., 6K) image reconstruction. To address these two challenges, we propose a novel framework (HyperThumbnail) for real-time 6K rate-distortion-aware image rescaling. Our framework first embeds an HR image into a JPEG LR thumbnail by an encoder with our proposed quantization prediction module, which minimizes the file size of the embedding LR JPEG thumbnail while maximizing HR reconstruction quality. Then, an efficient frequency-aware decoder reconstructs a high-fidelity HR image from the LR one in real time. Extensive experiments demonstrate that our framework outperforms previous image rescaling baselines in rate-distortion performance and can perform 6K image reconstruction in real time.

Related papers

• Realistic Extreme Image Rescaling via Generative Latent Space Learning [51.85790402171696]
  We propose a novel framework called Latent Space Based Image Rescaling (LSBIR) for extreme image rescaling tasks. LSBIR effectively leverages powerful natural image priors learned by a pre-trained text-to-image diffusion model to generate realistic HR images. In the first stage, a pseudo-invertible encoder-decoder models the bidirectional mapping between the latent features of the HR image and the target-sized LR image. In the second stage, the reconstructed features from the first stage are refined by a pre-trained diffusion model to generate more faithful and visually pleasing details.
  arXiv  Detail & Related papers  (2024-08-17T09:51:42Z)
• Self-Asymmetric Invertible Network for Compression-Aware Image Rescaling [6.861753163565238]
  In real-world applications, most images are compressed for transmission. We propose the Self-Asymmetric Invertible Network (SAIN) for compression-aware image rescaling.
  arXiv  Detail & Related papers  (2023-03-04T08:33:46Z)
• Real Image Super-Resolution using GAN through modeling of LR and HR process [20.537597542144916]
  We propose a learnable adaptive sinusoidal nonlinearities incorporated in LR and SR models by directly learn degradation distributions. We demonstrate the effectiveness of our proposed approach in quantitative and qualitative experiments.
  arXiv  Detail & Related papers  (2022-10-19T09:23:37Z)
• Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling [139.25215100378284]
  We propose a hierarchical conditional flow (HCFlow) as a unified framework for image SR and image rescaling. HCFlow learns a mapping between HR and LR image pairs by modelling the distribution of the LR image and the rest high-frequency component simultaneously. To further enhance the performance, other losses such as perceptual loss and GAN loss are combined with the commonly used negative log-likelihood loss in training.
  arXiv  Detail & Related papers  (2021-08-11T16:11:01Z)
• LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-Resolution and Beyond [75.37541439447314]
  Single image super-resolution (SISR) deals with a fundamental problem of upsampling a low-resolution (LR) image to its high-resolution (HR) version. This paper proposes a linearly-assembled pixel-adaptive regression network (LAPAR) to strike a sweet spot of deep model complexity and resulting SISR quality.
  arXiv  Detail & Related papers  (2021-05-21T15:47:18Z)
• Real-World Super-Resolution of Face-Images from Surveillance Cameras [25.258587196435464]
  We propose a novel framework for generation of realistic LR/HR training pairs. Our framework estimates realistic blur kernels, noise distributions, and JPEG compression artifacts to generate LR images with similar image characteristics as the ones in the source domain. For better perceptual quality we use a Generative Adrial Network (GAN) based SR model where we have exchanged the commonly used VGG-loss [24] with LPIPS-loss [52]
  arXiv  Detail & Related papers  (2021-02-05T11:38:30Z)
• Super-Resolution of Real-World Faces [3.4376560669160394]
  Real low-resolution (LR) face images contain degradations which are too varied and complex to be captured by known downsampling kernels. In this paper, we propose a two module super-resolution network where the feature extractor module extracts robust features from the LR image. We train a degradation GAN to convert bicubically downsampled clean images to real degraded images, and interpolate between the obtained degraded LR image and its clean LR counterpart.
  arXiv  Detail & Related papers  (2020-11-04T17:25:54Z)
• Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolution [73.86924594746884]
  Deep neural networks have exhibited promising performance in image super-resolution. These networks learn a nonlinear mapping function from low-resolution (LR) images to high-resolution (HR) images. We propose a dual regression scheme by introducing an additional constraint on LR data to reduce the space of the possible functions.
  arXiv  Detail & Related papers  (2020-03-16T04:23:42Z)
• PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models [77.32079593577821]
  PULSE (Photo Upsampling via Latent Space Exploration) generates high-resolution, realistic images at resolutions previously unseen in the literature. Our method outperforms state-of-the-art methods in perceptual quality at higher resolutions and scale factors than previously possible.
  arXiv  Detail & Related papers  (2020-03-08T16:44:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.