Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation

• URL: http://arxiv.org/abs/2409.09497v1
• Date: Sat, 14 Sep 2024 17:52:59 GMT
• Title: Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation
• Authors: Hugo Porta, Emanuele Dalsasso, Diego Marcos, Devis Tuia,
• Abstract summary: Prototypical part learning is emerging as a promising approach for making semantic segmentation interpretable. We propose a method for interpretable semantic segmentation that leverages multi-scale image representation for prototypical part learning. Experiments conducted on Pascal VOC, Cityscapes, and ADE20K demonstrate that the proposed method increases model sparsity, improves interpretability over existing prototype-based methods, and narrows the performance gap with the non-interpretable counterpart models.
• Score: 7.372346036256517
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Prototypical part learning is emerging as a promising approach for making semantic segmentation interpretable. The model selects real patches seen during training as prototypes and constructs the dense prediction map based on the similarity between parts of the test image and the prototypes. This improves interpretability since the user can inspect the link between the predicted output and the patterns learned by the model in terms of prototypical information. In this paper, we propose a method for interpretable semantic segmentation that leverages multi-scale image representation for prototypical part learning. First, we introduce a prototype layer that explicitly learns diverse prototypical parts at several scales, leading to multi-scale representations in the prototype activation output. Then, we propose a sparse grouping mechanism that produces multi-scale sparse groups of these scale-specific prototypical parts. This provides a deeper understanding of the interactions between multi-scale object representations while enhancing the interpretability of the segmentation model. The experiments conducted on Pascal VOC, Cityscapes, and ADE20K demonstrate that the proposed method increases model sparsity, improves interpretability over existing prototype-based methods, and narrows the performance gap with the non-interpretable counterpart models. Code is available at github.com/eceo-epfl/ScaleProtoSeg.

Related papers

• Interpretable Image Classification with Adaptive Prototype-based Vision Transformers [37.62530032165594]
  We present ProtoViT, a method for interpretable image classification combining deep learning and case-based reasoning. Our model integrates Vision Transformer (ViT) backbones into prototype based models, while offering spatially deformed prototypes. Our experiments show that our model can generally achieve higher performance than the existing prototype based models.
  arXiv  Detail & Related papers  (2024-10-28T04:33:28Z)
• ProtoSeg: Interpretable Semantic Segmentation with Prototypical Parts [12.959270094693254]
  We introduce ProtoSeg, a novel model for interpretable semantic image segmentation. To achieve accuracy comparable to baseline methods, we adapt the mechanism of prototypical parts. We show that ProtoSeg discovers semantic concepts, in contrast to standard segmentation models.
  arXiv  Detail & Related papers  (2023-01-28T19:14:32Z)
• Few-Shot Segmentation via Rich Prototype Generation and Recurrent Prediction Enhancement [12.614578133091168]
  We propose a rich prototype generation module (RPGM) and a recurrent prediction enhancement module (RPEM) to reinforce the prototype learning paradigm. RPGM combines superpixel and K-means clustering to generate rich prototype features with complementary scale relationships. RPEM utilizes the recurrent mechanism to design a round-way propagation decoder.
  arXiv  Detail & Related papers  (2022-10-03T08:46:52Z)
• Rethinking Semantic Segmentation: A Prototype View [126.59244185849838]
  We present a nonparametric semantic segmentation model based on non-learnable prototypes. Our framework yields compelling results over several datasets. We expect this work will provoke a rethink of the current de facto semantic segmentation model design.
  arXiv  Detail & Related papers  (2022-03-28T21:15:32Z)
• Dual Prototypical Contrastive Learning for Few-shot Semantic Segmentation [55.339405417090084]
  We propose a dual prototypical contrastive learning approach tailored to the few-shot semantic segmentation (FSS) task. The main idea is to encourage the prototypes more discriminative by increasing inter-class distance while reducing intra-class distance in prototype feature space. We demonstrate that the proposed dual contrastive learning approach outperforms state-of-the-art FSS methods on PASCAL-5i and COCO-20i datasets.
  arXiv  Detail & Related papers  (2021-11-09T08:14:50Z)
• Attentional Prototype Inference for Few-Shot Segmentation [128.45753577331422]
  We propose attentional prototype inference (API), a probabilistic latent variable framework for few-shot segmentation. We define a global latent variable to represent the prototype of each object category, which we model as a probabilistic distribution. We conduct extensive experiments on four benchmarks, where our proposal obtains at least competitive and often better performance than state-of-the-art prototype-based methods.
  arXiv  Detail & Related papers  (2021-05-14T06:58:44Z)
• SCNet: Enhancing Few-Shot Semantic Segmentation by Self-Contrastive Background Prototypes [56.387647750094466]
  Few-shot semantic segmentation aims to segment novel-class objects in a query image with only a few annotated examples. Most of advanced solutions exploit a metric learning framework that performs segmentation through matching each pixel to a learned foreground prototype. This framework suffers from biased classification due to incomplete construction of sample pairs with the foreground prototype only.
  arXiv  Detail & Related papers  (2021-04-19T11:21:47Z)
• Prototypical Representation Learning for Relation Extraction [56.501332067073065]
  This paper aims to learn predictive, interpretable, and robust relation representations from distantly-labeled data. We learn prototypes for each relation from contextual information to best explore the intrinsic semantics of relations. Results on several relation learning tasks show that our model significantly outperforms the previous state-of-the-art relational models.
  arXiv  Detail & Related papers  (2021-03-22T08:11:43Z)
• Part-aware Prototype Network for Few-shot Semantic Segmentation [50.581647306020095]
  We propose a novel few-shot semantic segmentation framework based on the prototype representation. Our key idea is to decompose the holistic class representation into a set of part-aware prototypes. We develop a novel graph neural network model to generate and enhance the proposed part-aware prototypes.
  arXiv  Detail & Related papers  (2020-07-13T11:03:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.