Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation

• URL: http://arxiv.org/abs/2407.18568v2
• Date: Mon, 29 Jul 2024 02:05:19 GMT
• Title: Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation
• Authors: Jingjun Yi, Qi Bi, Hao Zheng, Haolan Zhan, Wei Ji, Yawen Huang, Yuexiang Li, Yefeng Zheng,
• Abstract summary: We present a novel Spectral-dEcomposed Token (SET) learning framework to advance the frontier. Particularly, the frozen VFM features are first decomposed into the phase and amplitude components in the frequency space. We develop an attention optimization method to bridge the gap between style-affected representation and static tokens during inference.
• Score: 38.0401463751139
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The rapid development of Vision Foundation Model (VFM) brings inherent out-domain generalization for a variety of down-stream tasks. Among them, domain generalized semantic segmentation (DGSS) holds unique challenges as the cross-domain images share common pixel-wise content information but vary greatly in terms of the style. In this paper, we present a novel Spectral-dEcomposed Token (SET) learning framework to advance the frontier. Delving into further than existing fine-tuning token & frozen backbone paradigm, the proposed SET especially focuses on the way learning style-invariant features from these learnable tokens. Particularly, the frozen VFM features are first decomposed into the phase and amplitude components in the frequency space, which mainly contain the information of content and style, respectively, and then separately processed by learnable tokens for task-specific information extraction. After the decomposition, style variation primarily impacts the token-based feature enhancement within the amplitude branch. To address this issue, we further develop an attention optimization method to bridge the gap between style-affected representation and static tokens during inference. Extensive cross-domain experiments show its state-of-the-art performance.

Related papers

• Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning [86.99944014645322]
  We introduce a novel framework, Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning. We decompose each query image into its high-frequency and low-frequency components, and parallel incorporate them into the feature embedding network. Our framework establishes new state-of-the-art results on multiple cross-domain few-shot learning benchmarks.
  arXiv  Detail & Related papers  (2024-11-03T04:02:35Z)
• A Spitting Image: Modular Superpixel Tokenization in Vision Transformers [0.0]
  Vision Transformer (ViT) architectures traditionally employ a grid-based approach to tokenization independent of the semantic content of an image. We propose a modular superpixel tokenization strategy which decouples tokenization and feature extraction.
  arXiv  Detail & Related papers  (2024-08-14T17:28:58Z)
• FANet: Feature Amplification Network for Semantic Segmentation in Cluttered Background [9.970265640589966]
  Existing deep learning approaches leave out the semantic cues that are crucial in semantic segmentation present in complex scenarios. We propose a feature amplification network (FANet) as a backbone network that incorporates semantic information using a novel feature enhancement module at multi-stages. Our experimental results demonstrate the state-of-the-art performance compared to existing methods.
  arXiv  Detail & Related papers  (2024-07-12T15:57:52Z)
• Language Guided Domain Generalized Medical Image Segmentation [68.93124785575739]
  Single source domain generalization holds promise for more reliable and consistent image segmentation across real-world clinical settings. We propose an approach that explicitly leverages textual information by incorporating a contrastive learning mechanism guided by the text encoder features. Our approach achieves favorable performance against existing methods in literature.
  arXiv  Detail & Related papers  (2024-04-01T17:48:15Z)
• Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation [59.37587762543934]
  This paper studies the problem of weakly open-vocabulary semantic segmentation (WOVSS) Existing methods suffer from a granularity inconsistency regarding the usage of group tokens. We propose the prototypical guidance network (PGSeg) that incorporates multi-modal regularization.
  arXiv  Detail & Related papers  (2023-10-29T13:18:00Z)
• Learning Content-enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentation [28.165600284392042]
  Domain-generalized urban-scene semantic segmentation (USSS) aims to learn generalized semantic predictions across diverse urban-scene styles. Existing approaches typically rely on convolutional neural networks (CNNs) to learn the content of urban scenes. We propose a Content-enhanced Mask TransFormer (CMFormer) for domain-generalized USSS.
  arXiv  Detail & Related papers  (2023-07-01T15:48:33Z)
• Domain Generalization via Shuffled Style Assembly for Face Anti-Spoofing [69.80851569594924]
  Generalizable face anti-spoofing (FAS) has drawn growing attention. In this work, we separate the complete representation into content and style ones. A novel Shuffled Style Assembly Network (SSAN) is proposed to extract and reassemble different content and style features.
  arXiv  Detail & Related papers  (2022-03-10T12:44:05Z)
• GaitStrip: Gait Recognition via Effective Strip-based Feature Representations and Multi-Level Framework [34.397404430838286]
  We present a strip-based multi-level gait recognition network, named GaitStrip, to extract comprehensive gait information at different levels. To be specific, our high-level branch explores the context of gait sequences and our low-level one focuses on detailed posture changes. Our GaitStrip achieves state-of-the-art performance in both normal walking and complex conditions.
  arXiv  Detail & Related papers  (2022-03-08T09:49:48Z)
• Self-Supervised Tuning for Few-Shot Segmentation [82.32143982269892]
  Few-shot segmentation aims at assigning a category label to each image pixel with few annotated samples. Existing meta-learning method tends to fail in generating category-specifically discriminative descriptor when the visual features extracted from support images are marginalized in embedding space. This paper presents an adaptive framework tuning, in which the distribution of latent features across different episodes is dynamically adjusted based on a self-segmentation scheme.
  arXiv  Detail & Related papers  (2020-04-12T03:53:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.