Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution

• URL: http://arxiv.org/abs/2501.06838v3
• Date: Thu, 27 Feb 2025 09:07:43 GMT
• Title: Generalized and Efficient 2D Gaussian Splatting for Arbitrary-scale Super-Resolution
• Authors: Du Chen, Liyi Chen, Zhengqiang Zhang, Lei Zhang,
• Abstract summary: Implicit Neural Representation (INR) has been successfully employed for Arbitrary-scale Super-Resolution (ASR)<n>We develop two novel techniques to generalize GS for ASR.<n>We implement an efficient differentiable 2D GPU/CUDA-based scale-awareization to render super-resolved images.
• Score: 10.074968164380314
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Implicit Neural Representation (INR) has been successfully employed for Arbitrary-scale Super-Resolution (ASR). However, INR-based models need to query the multi-layer perceptron module numerous times and render a pixel in each query, resulting in insufficient representation capability and computational efficiency. Recently, Gaussian Splatting (GS) has shown its advantages over INR in both visual quality and rendering speed in 3D tasks, which motivates us to explore whether GS can be employed for the ASR task. However, directly applying GS to ASR is exceptionally challenging because the original GS is an optimization-based method through overfitting each single scene, while in ASR we aim to learn a single model that can generalize to different images and scaling factors. We overcome these challenges by developing two novel techniques. Firstly, to generalize GS for ASR, we elaborately design an architecture to predict the corresponding image-conditioned Gaussians of the input low-resolution image in a feed-forward manner. Each Gaussian can fit the shape and direction of an area of complex textures, showing powerful representation capability. Secondly, we implement an efficient differentiable 2D GPU/CUDA-based scale-aware rasterization to render super-resolved images by sampling discrete RGB values from the predicted continuous Gaussians. Via end-to-end training, our optimized network, namely GSASR, can perform ASR for any image and unseen scaling factors. Extensive experiments validate the effectiveness of our proposed method. The code and models will be released.

Related papers

• EigenGS Representation: From Eigenspace to Gaussian Image Space [20.454762899389358]
  EigenGS is an efficient transformation pipeline connecting eigenspace and image-space Gaussian representations. We show that EigenGS achieves superior reconstruction quality compared to direct 2D Gaussian fitting. The results highlight EigenGS's effectiveness and generalization ability across images with varying resolutions and diverse categories.
  arXiv  Detail & Related papers  (2025-03-10T15:27:03Z)
• Pixel to Gaussian: Ultra-Fast Continuous Super-Resolution with 2D Gaussian Modeling [50.34513854725803]
  Arbitrary-scale super-resolution (ASSR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs with arbitrary upsampling factors. We propose a novel ContinuousSR framework with a Pixel-to-Gaussian paradigm, which explicitly reconstructs 2D continuous HR signals from LR images using Gaussian Splatting.
  arXiv  Detail & Related papers  (2025-03-09T13:43:57Z)
• Large Images are Gaussians: High-Quality Large Image Representation with Levels of 2D Gaussian Splatting [21.629316414488027]
  We present textbfLarge textbfImages are textbfGaussians (textbfLIG), which delves deeper into the application of 2DGS for image representations.
  arXiv  Detail & Related papers  (2025-02-13T07:48:56Z)
• PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views [116.10577967146762]
  PixelGaussian is an efficient framework for learning generalizable 3D Gaussian reconstruction from arbitrary views. Our method achieves state-of-the-art performance with good generalization to various numbers of views.
  arXiv  Detail & Related papers  (2024-10-24T17:59:58Z)
• GaussianSR: High Fidelity 2D Gaussian Splatting for Arbitrary-Scale Image Super-Resolution [29.49617080140511]
  Implicit neural representations (INRs) have significantly advanced the field of arbitrary-scale super-resolution (ASSR) of images. Most existing INR-based ASSR networks first extract features from the given low-resolution image using an encoder, and then render the super-resolved result via a multi-layer perceptron decoder. We propose a novel ASSR method named GaussianSR that overcomes this limitation through 2D Gaussian Splatting (2DGS)
  arXiv  Detail & Related papers  (2024-07-25T13:53:48Z)
• Image-GS: Content-Adaptive Image Representation via 2D Gaussians [55.15950594752051]
  We propose Image-GS, a content-adaptive image representation. Using anisotropic 2D Gaussians as the basis, Image-GS shows high memory efficiency, supports fast random access, and offers a natural level of detail stack. General efficiency and fidelity of Image-GS are validated against several recent neural image representations and industry-standard texture compressors. We hope this research offers insights for developing new applications that require adaptive quality and resource control, such as machine perception, asset streaming, and content generation.
  arXiv  Detail & Related papers  (2024-07-02T00:45:21Z)
• GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering [112.16239342037714]
  GES (Generalized Exponential Splatting) is a novel representation that employs Generalized Exponential Function (GEF) to model 3D scenes. With the aid of a frequency-modulated loss, GES achieves competitive performance in novel-view synthesis benchmarks.
  arXiv  Detail & Related papers  (2024-02-15T17:32:50Z)
• GS-IR: 3D Gaussian Splatting for Inverse Rendering [71.14234327414086]
  We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian Splatting (GS) We extend GS, a top-performance representation for novel view synthesis, to estimate scene geometry, surface material, and environment illumination from multi-view images captured under unknown lighting conditions. The flexible and expressive GS representation allows us to achieve fast and compact geometry reconstruction, photorealistic novel view synthesis, and effective physically-based rendering.
  arXiv  Detail & Related papers  (2023-11-26T02:35:09Z)
• Effective Invertible Arbitrary Image Rescaling [77.46732646918936]
  Invertible Neural Networks (INN) are able to increase upscaling accuracy significantly by optimizing the downscaling and upscaling cycle jointly. A simple and effective invertible arbitrary rescaling network (IARN) is proposed to achieve arbitrary image rescaling by training only one model in this work. It is shown to achieve a state-of-the-art (SOTA) performance in bidirectional arbitrary rescaling without compromising perceptual quality in LR outputs.
  arXiv  Detail & Related papers  (2022-09-26T22:22:30Z)
• LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-Resolution and Beyond [75.37541439447314]
  Single image super-resolution (SISR) deals with a fundamental problem of upsampling a low-resolution (LR) image to its high-resolution (HR) version. This paper proposes a linearly-assembled pixel-adaptive regression network (LAPAR) to strike a sweet spot of deep model complexity and resulting SISR quality.
  arXiv  Detail & Related papers  (2021-05-21T15:47:18Z)
• SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models [19.17571465274627]
  Single image super-resolution (SISR) aims to reconstruct high-resolution (HR) images from the given low-resolution (LR) ones. We propose a novel single image super-resolution diffusion probabilistic model (SRDiff) SRDiff is optimized with a variant of the variational bound on the data likelihood and can provide diverse and realistic SR predictions.
  arXiv  Detail & Related papers  (2021-04-30T12:31:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.