When Semantic Segmentation Meets Frequency Aliasing

• URL: http://arxiv.org/abs/2403.09065v3
• Date: Mon, 25 Mar 2024 03:04:44 GMT
• Title: When Semantic Segmentation Meets Frequency Aliasing
• Authors: Linwei Chen, Lin Gu, Ying Fu,
• Abstract summary: We conduct a comprehensive analysis of hard pixel errors, categorizing them into three types: false responses, merging mistakes, and displacements. Our findings reveal a quantitative association between hard pixels and aliasing, which is distortion caused by the overlapping of frequency components in the Fourier domain during downsampling. Here, we propose two novel de-aliasing filter (DAF) and frequency mixing (FreqMix) modules to alleviate aliasing by accurately removing or adjusting frequencies higher than the Nyquist frequency.
• Score: 14.066404173580864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite recent advancements in semantic segmentation, where and what pixels are hard to segment remains largely unexplored. Existing research only separates an image into easy and hard regions and empirically observes the latter are associated with object boundaries. In this paper, we conduct a comprehensive analysis of hard pixel errors, categorizing them into three types: false responses, merging mistakes, and displacements. Our findings reveal a quantitative association between hard pixels and aliasing, which is distortion caused by the overlapping of frequency components in the Fourier domain during downsampling. To identify the frequencies responsible for aliasing, we propose using the equivalent sampling rate to calculate the Nyquist frequency, which marks the threshold for aliasing. Then, we introduce the aliasing score as a metric to quantify the extent of aliasing. While positively correlated with the proposed aliasing score, three types of hard pixels exhibit different patterns. Here, we propose two novel de-aliasing filter (DAF) and frequency mixing (FreqMix) modules to alleviate aliasing degradation by accurately removing or adjusting frequencies higher than the Nyquist frequency. The DAF precisely removes the frequencies responsible for aliasing before downsampling, while the FreqMix dynamically selects high-frequency components within the encoder block. Experimental results demonstrate consistent improvements in semantic segmentation and low-light instance segmentation tasks. The code is available at: https://github.com/Linwei-Chen/Seg-Aliasing.

Related papers

• Freq-Mip-AA : Frequency Mip Representation for Anti-Aliasing Neural Radiance Fields [3.796287987989994]
  Mip-NeRF proposed using frustums to render a pixel and suggested integrated positional encoding (IPE) While effective, this approach requires long training times due to its reliance on volumetric architecture. We propose a novel anti-aliasing technique that utilizes grid-based representations, usually showing significantly faster training time.
  arXiv  Detail & Related papers  (2024-06-19T06:33:56Z)
• Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Learning [81.98675881423131]
  This research addresses the challenge of developing a universal deepfake detector that can effectively identify unseen deepfake images. Existing frequency-based paradigms have relied on frequency-level artifacts introduced during the up-sampling in GAN pipelines to detect forgeries. We introduce a novel frequency-aware approach called FreqNet, centered around frequency domain learning, specifically designed to enhance the generalizability of deepfake detectors.
  arXiv  Detail & Related papers  (2024-03-12T01:28:00Z)
• FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization [67.47895278233717]
  We develop a progressive frequency regularization technique to tackle the over-reconstruction issue within the frequency space. FreGS achieves superior novel view synthesis and outperforms the state-of-the-art consistently.
  arXiv  Detail & Related papers  (2024-03-11T17:00:27Z)
• FS-BAND: A Frequency-Sensitive Banding Detector [55.59101150019851]
  Banding artifact, as known as staircase-like contour, is a common quality annoyance that happens in compression, transmission, etc. We propose a no-reference banding detection model to capture and evaluate banding artifacts, called the Frequency-Sensitive BANding Detector (FS-BAND) Experimental results show that the proposed FS-BAND method outperforms state-of-the-art image quality assessment (IQA) approaches with higher accuracy in banding classification task.
  arXiv  Detail & Related papers  (2023-11-30T03:20:42Z)
• Improving Feature Stability during Upsampling -- Spectral Artifacts and the Importance of Spatial Context [15.351461000403074]
  Pixel-wise predictions are required in a wide variety of tasks such as image restoration, image segmentation, or disparity estimation. Previous works have shown that resampling operations are subject to artifacts such as aliasing. We show that the availability of large spatial context during upsampling allows to provide stable, high-quality pixel-wise predictions.
  arXiv  Detail & Related papers  (2023-11-29T10:53:05Z)
• Fix your downsampling ASAP! Be natively more robust via Aliasing and Spectral Artifact free Pooling [11.72025865314187]
  Convolutional neural networks encode images through a sequence of convolutions, normalizations and non-linearities as well as downsampling operations. Previous work showed that even slight mistakes during sampling, leading to aliasing, can be directly attributed to the networks' lack in robustness. We propose aliasing and spectral artifact-free pooling, short ASAP.
  arXiv  Detail & Related papers  (2023-07-19T07:47:23Z)
• Category-Adaptive Label Discovery and Noise Rejection for Multi-label Image Recognition with Partial Positive Labels [78.88007892742438]
  Training multi-label models with partial positive labels (MLR-PPL) attracts increasing attention. Previous works regard unknown labels as negative and adopt traditional MLR algorithms. We propose to explore semantic correlation among different images to facilitate the MLR-PPL task.
  arXiv  Detail & Related papers  (2022-11-15T02:11:20Z)
• Adaptive Frequency Learning in Two-branch Face Forgery Detection [66.91715092251258]
  We propose Adaptively learn Frequency information in the two-branch Detection framework, dubbed AFD. We liberate our network from the fixed frequency transforms, and achieve better performance with our data- and task-dependent transform layers.
  arXiv  Detail & Related papers  (2022-03-27T14:25:52Z)
• Frequency Pooling: Shift-Equivalent and Anti-Aliasing Downsampling [9.249235534786072]
  We show that frequency pooling is shift-equivalent and anti-aliasing based on the property of Fourier transform and Nyquist frequency. Experiments on image classification show that frequency pooling improves accuracy and robustness with respect to the shifts of CNNs.
  arXiv  Detail & Related papers  (2021-09-24T09:32:10Z)
• Wavelet-Based Network For High Dynamic Range Imaging [64.66969585951207]
  Existing methods, such as optical flow based and end-to-end deep learning based solutions, are error-prone either in detail restoration or ghosting artifacts removal. In this work, we propose a novel frequency-guided end-to-end deep neural network (FNet) to conduct HDR fusion in the frequency domain, and Wavelet Transform (DWT) is used to decompose inputs into different frequency bands. The low-frequency signals are used to avoid specific ghosting artifacts, while the high-frequency signals are used for preserving details.
  arXiv  Detail & Related papers  (2021-08-03T12:26:33Z)
• Low Pass Filter for Anti-aliasing in Temporal Action Localization [15.139834271977913]
  This paper aims to verify the existence of aliasing in temporal action localization methods. It investigates utilizing low pass filters to solve this problem by inhibiting the high-frequency band. Experiments demonstrate that anti-aliasing with low pass filters in TAL is advantageous and efficient.
  arXiv  Detail & Related papers  (2021-04-23T03:57:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.