Unsupervised Segmentation by Diffusing, Walking and Cutting

• URL: http://arxiv.org/abs/2412.04678v1
• Date: Fri, 06 Dec 2024 00:23:18 GMT
• Title: Unsupervised Segmentation by Diffusing, Walking and Cutting
• Authors: Daniela Ivanova, Marco Aversa, Paul Henderson, John Williamson,
• Abstract summary: We propose an unsupervised image segmentation method using features from pre-trained text-to-image diffusion models. A key insight is that self-attention probability distributions can be interpreted as a transition matrix for random walks across the image. We show that our approach surpasses all existing methods for zero-shot unsupervised segmentation, achieving state-of-the-art results on COCO-Stuff-27 and Cityscapes.
• Score: 5.6872893893453105
• License:
• Abstract: We propose an unsupervised image segmentation method using features from pre-trained text-to-image diffusion models. Inspired by classic spectral clustering approaches, we construct adjacency matrices from self-attention layers between image patches and recursively partition using Normalised Cuts. A key insight is that self-attention probability distributions, which capture semantic relations between patches, can be interpreted as a transition matrix for random walks across the image. We leverage this by first using Random Walk Normalized Cuts directly on these self-attention activations to partition the image, minimizing transition probabilities between clusters while maximizing coherence within clusters. Applied recursively, this yields a hierarchical segmentation that reflects the rich semantics in the pre-trained attention layers, without any additional training. Next, we explore other ways to build the NCuts adjacency matrix from features, and how we can use the random walk interpretation of self-attention to capture long-range relationships. Finally, we propose an approach to automatically determine the NCut cost criterion, avoiding the need to tune this manually. We quantitatively analyse the effect incorporating different features, a constant versus dynamic NCut threshold, and incorporating multi-node paths when constructing the NCuts adjacency matrix. We show that our approach surpasses all existing methods for zero-shot unsupervised segmentation, achieving state-of-the-art results on COCO-Stuff-27 and Cityscapes.

Related papers

• Freestyle Sketch-in-the-Loop Image Segmentation [116.1810651297801]
  We introduce a "sketch-in-the-loop" image segmentation framework, enabling the segmentation of visual concepts partially, completely, or in groupings. This framework capitalises on the synergy between sketch-based image retrieval models and large-scale pre-trained models. Our purpose-made augmentation strategy enhances the versatility of our sketch-guided mask generation, allowing segmentation at multiple levels.
  arXiv  Detail & Related papers  (2025-01-27T13:07:51Z)
• Unsupervised Representation Learning by Balanced Self Attention Matching [2.3020018305241337]
  We present a self-supervised method for embedding image features called BAM. We obtain rich representations and avoid feature collapse by minimizing a loss that matches these distributions to their globally balanced and entropy regularized version. We show competitive performance with leading methods on both semi-supervised and transfer-learning benchmarks.
  arXiv  Detail & Related papers  (2024-08-04T12:52:44Z)
• Learning to Rank Patches for Unbiased Image Redundancy Reduction [80.93989115541966]
  Images suffer from heavy spatial redundancy because pixels in neighboring regions are spatially correlated. Existing approaches strive to overcome this limitation by reducing less meaningful image regions. We propose a self-supervised framework for image redundancy reduction called Learning to Rank Patches.
  arXiv  Detail & Related papers  (2024-03-31T13:12:41Z)
• Self-supervised Few-shot Learning for Semantic Segmentation: An Annotation-free Approach [4.855689194518905]
  Few-shot semantic segmentation (FSS) offers immense potential in the field of medical image analysis. Existing FSS techniques heavily rely on annotated semantic classes, rendering them unsuitable for medical images. We propose a novel self-supervised FSS framework that does not rely on any annotation. Instead, it adaptively estimates the query mask by leveraging the eigenvectors obtained from the support images.
  arXiv  Detail & Related papers  (2023-07-26T18:33:30Z)
• DeepCut: Unsupervised Segmentation using Graph Neural Networks Clustering [6.447863458841379]
  This study introduces a lightweight Graph Neural Network (GNN) to replace classical clustering methods. Unlike existing methods, our GNN takes both the pair-wise affinities between local image features and the raw features as input. We demonstrate how classical clustering objectives can be formulated as self-supervised loss functions for training an image segmentation GNN.
  arXiv  Detail & Related papers  (2022-12-12T12:31:46Z)
• LEAD: Self-Supervised Landmark Estimation by Aligning Distributions of Feature Similarity [49.84167231111667]
  Existing works in self-supervised landmark detection are based on learning dense (pixel-level) feature representations from an image. We introduce an approach to enhance the learning of dense equivariant representations in a self-supervised fashion. We show that having such a prior in the feature extractor helps in landmark detection, even under drastically limited number of annotations.
  arXiv  Detail & Related papers  (2022-04-06T17:48:18Z)
• Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences [118.6018141306409]
  We propose Probabilistic Warp Consistency, a weakly-supervised learning objective for semantic matching. We first construct an image triplet by applying a known warp to one of the images in a pair depicting different instances of the same object class. Our objective also brings substantial improvements in the strongly-supervised regime, when combined with keypoint annotations.
  arXiv  Detail & Related papers  (2022-03-08T18:55:11Z)
• Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation [49.90178055521207]
  This work addresses weakly supervised semantic segmentation (WSSS), with the goal of bridging the gap between image-level annotations and pixel-level segmentation. We formulate WSSS as a novel group-wise learning task that explicitly models semantic dependencies in a group of images to estimate more reliable pseudo ground-truths. In particular, we devise a graph neural network (GNN) for group-wise semantic mining, wherein input images are represented as graph nodes.
  arXiv  Detail & Related papers  (2020-12-09T12:40:13Z)
• Unsupervised Learning of Image Segmentation Based on Differentiable Feature Clustering [14.074732867392008]
  The usage of convolutional neural networks (CNNs) for unsupervised image segmentation was investigated in this study. We present a novel end-to-end network of unsupervised image segmentation that consists of normalization and an argmax function for differentiable clustering. Third, we present an extension of the proposed method for segmentation with scribbles as user input, which showed better accuracy than existing methods.
  arXiv  Detail & Related papers  (2020-07-20T10:28:36Z)
• Self-Supervised Tuning for Few-Shot Segmentation [82.32143982269892]
  Few-shot segmentation aims at assigning a category label to each image pixel with few annotated samples. Existing meta-learning method tends to fail in generating category-specifically discriminative descriptor when the visual features extracted from support images are marginalized in embedding space. This paper presents an adaptive framework tuning, in which the distribution of latent features across different episodes is dynamically adjusted based on a self-segmentation scheme.
  arXiv  Detail & Related papers  (2020-04-12T03:53:53Z)
• GATCluster: Self-Supervised Gaussian-Attention Network for Image Clustering [9.722607434532883]
  We propose a self-supervised clustering network for image Clustering (GATCluster) Rather than extracting intermediate features first and then performing the traditional clustering, GATCluster semantic cluster labels without further post-processing. We develop a two-step learning algorithm that is memory-efficient for clustering large-size images.
  arXiv  Detail & Related papers  (2020-02-27T00:57:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.