SegDiscover: Visual Concept Discovery via Unsupervised Semantic Segmentation

• URL: http://arxiv.org/abs/2204.10926v1
• Date: Fri, 22 Apr 2022 20:44:42 GMT
• Title: SegDiscover: Visual Concept Discovery via Unsupervised Semantic Segmentation
• Authors: Haiyang Huang, Zhi Chen, Cynthia Rudin
• Abstract summary: SegDiscover is a novel framework that discovers semantically meaningful visual concepts from imagery datasets with complex scenes without supervision. Our method generates concept primitives from raw images, discovering concepts by clustering in the latent space of a self-supervised pretrained encoder, and concept refinement via neural network smoothing.
• Score: 29.809900593362844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual concept discovery has long been deemed important to improve interpretability of neural networks, because a bank of semantically meaningful concepts would provide us with a starting point for building machine learning models that exhibit intelligible reasoning process. Previous methods have disadvantages: either they rely on labelled support sets that incorporate human biases for objects that are "useful," or they fail to identify multiple concepts that occur within a single image. We reframe the concept discovery task as an unsupervised semantic segmentation problem, and present SegDiscover, a novel framework that discovers semantically meaningful visual concepts from imagery datasets with complex scenes without supervision. Our method contains three important pieces: generating concept primitives from raw images, discovering concepts by clustering in the latent space of a self-supervised pretrained encoder, and concept refinement via neural network smoothing. Experimental results provide evidence that our method can discover multiple concepts within a single image and outperforms state-of-the-art unsupervised methods on complex datasets such as Cityscapes and COCO-Stuff. Our method can be further used as a neural network explanation tool by comparing results obtained by different encoders.

Related papers

• Exploiting Interpretable Capabilities with Concept-Enhanced Diffusion and Prototype Networks [8.391254800873599]
  We create concept-enriched models that incorporate concept information into existing architectures. In particular, we propose Concept-Guided Diffusion Conditional, which can generate visual representations of concepts, and Concept-Guided Prototype Networks, which can create a concept prototype dataset and leverage it to perform interpretable concept prediction. These results open up new lines of research by exploiting pre-existing information in the quest for rendering machine learning more human-understandable.
  arXiv  Detail & Related papers  (2024-10-24T13:07:56Z)
• Explainable Concept Generation through Vision-Language Preference Learning [7.736445799116692]
  Concept-based explanations have become a popular choice for explaining deep neural networks post-hoc. We devise a reinforcement learning-based preference optimization algorithm that fine-tunes the vision-language generative model. In addition to showing the efficacy and reliability of our method, we show how our method can be used as a diagnostic tool for analyzing neural networks.
  arXiv  Detail & Related papers  (2024-08-24T02:26:42Z)
• Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery [52.498055901649025]
  Concept Bottleneck Models (CBMs) have been proposed to address the 'black-box' problem of deep neural networks. We propose a novel CBM approach -- called Discover-then-Name-CBM (DN-CBM) -- that inverts the typical paradigm. Our concept extraction strategy is efficient, since it is agnostic to the downstream task, and uses concepts already known to the model.
  arXiv  Detail & Related papers  (2024-07-19T17:50:11Z)
• LLM-assisted Concept Discovery: Automatically Identifying and Explaining Neuron Functions [15.381209058506078]
  Prior works have associated concepts with neurons based on examples of concepts or a pre-defined set of concepts. We propose to leverage multimodal large language models for automatic and open-ended concept discovery. We validate each concept by generating examples and counterexamples and evaluating the neuron's response on this new set of images.
  arXiv  Detail & Related papers  (2024-06-12T18:19:37Z)
• CLiC: Concept Learning in Context [54.81654147248919]
  This paper builds upon recent advancements in visual concept learning. It involves acquiring a visual concept from a source image and subsequently applying it to an object in a target image. To localize the concept learning, we employ soft masks that contain both the concept within the mask and the surrounding image area.
  arXiv  Detail & Related papers  (2023-11-28T01:33:18Z)
• Concept Decomposition for Visual Exploration and Inspiration [53.06983340652571]
  We propose a method to decompose a visual concept into different visual aspects encoded in a hierarchical tree structure. We utilize large vision-language models and their rich latent space for concept decomposition and generation.
  arXiv  Detail & Related papers  (2023-05-29T16:56:56Z)
• Visual Superordinate Abstraction for Robust Concept Learning [80.15940996821541]
  Concept learning constructs visual representations that are connected to linguistic semantics. We ascribe the bottleneck to a failure of exploring the intrinsic semantic hierarchy of visual concepts. We propose a visual superordinate abstraction framework for explicitly modeling semantic-aware visual subspaces.
  arXiv  Detail & Related papers  (2022-05-28T14:27:38Z)
• Visual Concepts Tokenization [65.61987357146997]
  We propose an unsupervised transformer-based Visual Concepts Tokenization framework, dubbed VCT, to perceive an image into a set of disentangled visual concept tokens. To obtain these concept tokens, we only use cross-attention to extract visual information from the image tokens layer by layer without self-attention between concept tokens. We further propose a Concept Disentangling Loss to facilitate that different concept tokens represent independent visual concepts.
  arXiv  Detail & Related papers  (2022-05-20T11:25:31Z)
• Interactive Disentanglement: Learning Concepts by Interacting with their Prototype Representations [15.284688801788912]
  We show the advantages of prototype representations for understanding and revising the latent space of neural concept learners. For this purpose, we introduce interactive Concept Swapping Networks (iCSNs) iCSNs learn to bind conceptual information to specific prototype slots by swapping the latent representations of paired images.
  arXiv  Detail & Related papers  (2021-12-04T09:25:40Z)
• Unsupervised Learning of Compositional Energy Concepts [70.11673173291426]
  We propose COMET, which discovers and represents concepts as separate energy functions. Comet represents both global concepts as well as objects under a unified framework.
  arXiv  Detail & Related papers  (2021-11-04T17:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.