Improving Feature Stability during Upsampling -- Spectral Artifacts and the Importance of Spatial Context

• URL: http://arxiv.org/abs/2311.17524v2
• Date: Fri, 12 Jul 2024 20:58:59 GMT
• Title: Improving Feature Stability during Upsampling -- Spectral Artifacts and the Importance of Spatial Context
• Authors: Shashank Agnihotri, Julia Grabinski, Margret Keuper,
• Abstract summary: Pixel-wise predictions are required in a wide variety of tasks such as image restoration, image segmentation, or disparity estimation. Previous works have shown that resampling operations are subject to artifacts such as aliasing. We show that the availability of large spatial context during upsampling allows to provide stable, high-quality pixel-wise predictions.
• Score: 15.351461000403074
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pixel-wise predictions are required in a wide variety of tasks such as image restoration, image segmentation, or disparity estimation. Common models involve several stages of data resampling, in which the resolution of feature maps is first reduced to aggregate information and then increased to generate a high-resolution output. Previous works have shown that resampling operations are subject to artifacts such as aliasing. During downsampling, aliases have been shown to compromise the prediction stability of image classifiers. During upsampling, they have been leveraged to detect generated content. Yet, the effect of aliases during upsampling has not yet been discussed w.r.t. the stability and robustness of pixel-wise predictions. While falling under the same term (aliasing), the challenges for correct upsampling in neural networks differ significantly from those during downsampling: when downsampling, some high frequencies can not be correctly represented and have to be removed to avoid aliases. However, when upsampling for pixel-wise predictions, we actually require the model to restore such high frequencies that can not be encoded in lower resolutions. The application of findings from signal processing is therefore a necessary but not a sufficient condition to achieve the desirable output. In contrast, we find that the availability of large spatial context during upsampling allows to provide stable, high-quality pixel-wise predictions, even when fully learning all filter weights.

Related papers

• HyperSpace: Hypernetworks for spacing-adaptive image segmentation [0.05958478403940788]
  We propose to condition segmentation models on the voxel spacing using hypernetworks. Our approach allows processing images at their native resolutions or at resolutions adjusted to the hardware and time constraints at inference time.
  arXiv  Detail & Related papers  (2024-07-04T07:09:23Z)
• Towards Efficient and Accurate CT Segmentation via Edge-Preserving Probabilistic Downsampling [2.1465347972460367]
  Downsampling images and labels, often necessitated by limited resources or to expedite network training, leads to the loss of small objects and thin boundaries. This undermines the segmentation network's capacity to interpret images accurately and predict detailed labels, resulting in diminished performance compared to processing at original resolutions. We introduce a novel method named Edge-preserving Probabilistic Downsampling (EPD) It utilizes class uncertainty within a local window to produce soft labels, with the window size dictating the downsampling factor.
  arXiv  Detail & Related papers  (2024-04-05T10:01:31Z)
• Fix your downsampling ASAP! Be natively more robust via Aliasing and Spectral Artifact free Pooling [11.72025865314187]
  Convolutional neural networks encode images through a sequence of convolutions, normalizations and non-linearities as well as downsampling operations. Previous work showed that even slight mistakes during sampling, leading to aliasing, can be directly attributed to the networks' lack in robustness. We propose aliasing and spectral artifact-free pooling, short ASAP.
  arXiv  Detail & Related papers  (2023-07-19T07:47:23Z)
• Grad-PU: Arbitrary-Scale Point Cloud Upsampling via Gradient Descent with Learned Distance Functions [77.32043242988738]
  We propose a new framework for accurate point cloud upsampling that supports arbitrary upsampling rates. Our method first interpolates the low-res point cloud according to a given upsampling rate.
  arXiv  Detail & Related papers  (2023-04-24T06:36:35Z)
• Towards Bidirectional Arbitrary Image Rescaling: Joint Optimization and Cycle Idempotence [76.93002743194974]
  We propose a method to treat arbitrary rescaling, both upscaling and downscaling, as one unified process. The proposed model is able to learn upscaling and downscaling simultaneously and achieve bidirectional arbitrary image rescaling. It is shown to be robust in cycle idempotence test, free of severe degradations in reconstruction accuracy when the downscaling-to-upscaling cycle is applied repetitively.
  arXiv  Detail & Related papers  (2022-03-02T07:42:15Z)
• Upsampling layers for music source separation [12.982998040587665]
  Upsampling artifacts can either be tonal artifacts (additive high-frequency noise) or filtering artifacts (substractive, attenuating some bands) We study how different artifacts interact and assess their impact on the models' performance. Our results show that filtering artifacts, associated with upsamplers, are perceptually preferable, even if they tend to achieve worse objective scores.
  arXiv  Detail & Related papers  (2021-11-23T10:36:28Z)
• Toward Real-World Super-Resolution via Adaptive Downsampling Models [58.38683820192415]
  This study proposes a novel method to simulate an unknown downsampling process without imposing restrictive prior knowledge. We propose a generalizable low-frequency loss (LFL) in the adversarial training framework to imitate the distribution of target LR images without using any paired examples.
  arXiv  Detail & Related papers  (2021-09-08T06:00:32Z)
• Designing a Practical Degradation Model for Deep Blind Image Super-Resolution [134.9023380383406]
  Single image super-resolution (SISR) methods would not perform well if the assumed degradation model deviates from those in real images. This paper proposes to design a more complex but practical degradation model that consists of randomly shuffled blur, downsampling and noise degradations.
  arXiv  Detail & Related papers  (2021-03-25T17:40:53Z)
• Scale factor point spread function matching: Beyond aliasing in image resampling [4.81150027600776]
  Imaging devices exploit the Nyquist-Shannon sampling theorem to avoid both aliasing and redundant oversampling by design. In medical image resampling, images are considered as continuous functions, are warped by a spatial transformation, and are then sampled on a regular grid. This paper shows that this oversight introduces artefacts, including aliasing, that can lead to important bias in clinical applications.
  arXiv  Detail & Related papers  (2021-01-16T11:40:58Z)
• Learning Affinity-Aware Upsampling for Deep Image Matting [83.02806488958399]
  We show that learning affinity in upsampling provides an effective and efficient approach to exploit pairwise interactions in deep networks. In particular, results on the Composition-1k matting dataset show that A2U achieves a 14% relative improvement in the SAD metric against a strong baseline. Compared with the state-of-the-art matting network, we achieve 8% higher performance with only 40% model complexity.
  arXiv  Detail & Related papers  (2020-11-29T05:09:43Z)
• Invertible Image Rescaling [118.2653765756915]
  We develop an Invertible Rescaling Net (IRN) to produce visually-pleasing low-resolution images. We capture the distribution of the lost information using a latent variable following a specified distribution in the downscaling process.
  arXiv  Detail & Related papers  (2020-05-12T09:55:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.