iSeg: An Iterative Refinement-based Framework for Training-free Segmentation

• URL: http://arxiv.org/abs/2409.03209v4
• Date: Tue, 8 Oct 2024 08:32:36 GMT
• Title: iSeg: An Iterative Refinement-based Framework for Training-free Segmentation
• Authors: Lin Sun, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, Yanwei Pang,
• Abstract summary: We present a deep experimental analysis on iteratively refining cross-attention map with self-attention map. We propose an effective iterative refinement framework for training-free segmentation, named iSeg. Our proposed iSeg achieves an absolute gain of 3.8% in terms of mIoU compared to the best existing training-free approach in literature.
• Score: 85.58324416386375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Stable diffusion has demonstrated strong image synthesis ability to given text descriptions, suggesting it to contain strong semantic clue for grouping objects. The researchers have explored employing stable diffusion for training-free segmentation. Most existing approaches refine cross-attention map by self-attention map once, demonstrating that self-attention map contains useful semantic information to improve segmentation. To fully utilize self-attention map, we present a deep experimental analysis on iteratively refining cross-attention map with self-attention map, and propose an effective iterative refinement framework for training-free segmentation, named iSeg. The proposed iSeg introduces an entropy-reduced self-attention module that utilizes a gradient descent scheme to reduce the entropy of self-attention map, thereby suppressing the weak responses corresponding to irrelevant global information. Leveraging the entropy-reduced self-attention module, our iSeg stably improves refined cross-attention map with iterative refinement. Further, we design a category-enhanced cross-attention module to generate accurate cross-attention map, providing a better initial input for iterative refinement. Extensive experiments across different datasets and diverse segmentation tasks reveal the merits of proposed contributions, leading to promising performance on diverse segmentation tasks. For unsupervised semantic segmentation on Cityscapes, our iSeg achieves an absolute gain of 3.8% in terms of mIoU compared to the best existing training-free approach in literature. Moreover, our proposed iSeg can support segmentation with different kinds of images and interactions. The project is available at https://linsun449.github.io/iSeg.

Related papers

• ResCLIP: Residual Attention for Training-free Dense Vision-language Inference [27.551367463011008]
  Cross-correlation of self-attention in CLIP's non-final layers also exhibits localization properties. We propose the Residual Cross-correlation Self-attention (RCS) module, which leverages the cross-correlation self-attention from intermediate layers to remold the attention in the final block. The RCS module effectively reorganizes spatial information, unleashing the localization potential within CLIP for dense vision-language inference.
  arXiv  Detail & Related papers  (2024-11-24T14:14:14Z)
• Auxiliary Tasks Enhanced Dual-affinity Learning for Weakly Supervised Semantic Segmentation [79.05949524349005]
  We propose AuxSegNet+, a weakly supervised auxiliary learning framework to explore the rich information from saliency maps. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps.
  arXiv  Detail & Related papers  (2024-03-02T10:03:21Z)
• DiffVein: A Unified Diffusion Network for Finger Vein Segmentation and Authentication [50.017055360261665]
  We introduce DiffVein, a unified diffusion model-based framework which simultaneously addresses vein segmentation and authentication tasks. For better feature interaction between these two branches, we introduce two specialized modules. In this way, our framework allows for a dynamic interplay between diffusion and segmentation embeddings.
  arXiv  Detail & Related papers  (2024-02-03T06:49:42Z)
• Self-supervised Few-shot Learning for Semantic Segmentation: An Annotation-free Approach [4.855689194518905]
  Few-shot semantic segmentation (FSS) offers immense potential in the field of medical image analysis. Existing FSS techniques heavily rely on annotated semantic classes, rendering them unsuitable for medical images. We propose a novel self-supervised FSS framework that does not rely on any annotation. Instead, it adaptively estimates the query mask by leveraging the eigenvectors obtained from the support images.
  arXiv  Detail & Related papers  (2023-07-26T18:33:30Z)
• Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentation [88.49669148290306]
  We propose a novel weakly supervised multi-task framework called AuxSegNet to leverage saliency detection and multi-label image classification as auxiliary tasks. Inspired by their similar structured semantics, we also propose to learn a cross-task global pixel-level affinity map from the saliency and segmentation representations. The learned cross-task affinity can be used to refine saliency predictions and propagate CAM maps to provide improved pseudo labels for both tasks.
  arXiv  Detail & Related papers  (2021-07-25T11:39:58Z)
• Self-Guided and Cross-Guided Learning for Few-Shot Segmentation [12.899804391102435]
  We propose a self-guided learning approach for few-shot segmentation. By making an initial prediction for the annotated support image, the covered and uncovered foreground regions are encoded to the primary and auxiliary support vectors. By aggregating both primary and auxiliary support vectors, better segmentation performances are obtained on query images.
  arXiv  Detail & Related papers  (2021-03-30T07:36:41Z)
• Bayesian Attention Modules [65.52970388117923]
  We propose a scalable version of attention that is easy to implement and optimize. Our experiments show the proposed method brings consistent improvements over the corresponding baselines.
  arXiv  Detail & Related papers  (2020-10-20T20:30:55Z)
• Mixup-CAM: Weakly-supervised Semantic Segmentation via Uncertainty Regularization [73.03956876752868]
  We propose a principled and end-to-end train-able framework to allow the network to pay attention to other parts of the object. Specifically, we introduce the mixup data augmentation scheme into the classification network and design two uncertainty regularization terms to better interact with the mixup strategy.
  arXiv  Detail & Related papers  (2020-08-03T21:19:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.