Explaining Image Similarity with Automatically Extracted Concept Activation Vectors

• URL: http://arxiv.org/abs/2607.28386v1
• Date: Thu, 30 Jul 2026 15:43:33 GMT
• Title: Explaining Image Similarity with Automatically Extracted Concept Activation Vectors
• Abstract summary: We introduce a model- and metric-agnostic framework that explains image similarity using Concept Activation Vectors (CAVs) extracted automatically via Sparse Autoencoders (SAEs)<n>We extend this procedure to group-level settings, explaining what drives similarity across a cluster of images rather than a single pair.<n>Our experiments show that our latent perturbations are more faithful to the underlying data distribution than pixel-space baselines.
• Score: 43.61120317216353
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image similarity underlies many computer vision applications, yet it is often unclear why two images receive a high or low similarity score. Existing explainability methods often rely on gradient-based attribution maps to provide local justifications for similarity. These approaches struggle to provide global insights into what specifically drives similarity in regions of an embedding space, such as texture, shape, or color. We introduce a model- and metric-agnostic framework that explains image similarity using Concept Activation Vectors (CAVs) extracted automatically via Sparse Autoencoders (SAEs). Given a pair of images, we perturb their embeddings along discovered concept directions and measure the resulting change in a chosen similarity function, yielding concept importances. For image pairs, we provide localization with concept attribution maps. We extend this procedure to group-level settings, explaining what drives similarity across a cluster of images rather than a single pair, and further, we introduce Exemplar Retrieval, aiming to recover samples with similar reasons contributing to similarity. Our experiments show that our latent perturbations are more faithful to the underlying data distribution than pixel-space baselines, and that concept importances linearly recover the true similarity score. Qualitative results further confirm the usefulness of our methods in understanding a model's individual and group similarity judgments.

Related papers

• Image Difference Quantification Using Autoencoder-Based Latent Representations [0.5352699766206808]
  This paper proposes a convolutional autoencoder-based framework for quantifying image differences using cosine similarity in latent space.<n>The proposed approach provides a computationally efficient and semantically grounded alternative to conventional pixel-based similarity metrics.
  arXiv  Detail & Related papers  (2026-08-25T16:23:47Z)
• Relational Visual Similarity [75.39827145344957]
  relational similarity is arguable by cognitive scientist to be what distinguishes humans from other species.<n>All widely used visual similarity metrics today focus solely on perceptual attribute similarity.<n>Our study shows that while relational similarity has a lot of real-world applications, existing image similarity models fail to capture it.
  arXiv  Detail & Related papers  (2025-12-08T18:59:56Z)
• Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding [112.0878081944858]
  Quantifying the degree of similarity between images is a key copyright issue for image-based machine learning. We seek to define and compute a notion of "conceptual similarity" among images that captures high-level relations. Two highly dissimilar images can be discriminated early in their description, whereas conceptually dissimilar ones will need more detail to be distinguished.
  arXiv  Detail & Related papers  (2024-02-14T03:31:17Z)
• Bayesian Unsupervised Disentanglement of Anatomy and Geometry for Deep Groupwise Image Registration [59.062085785106234]
  This article presents a general Bayesian learning framework for multi-modal groupwise image registration.<n>We propose a novel hierarchical variational auto-encoding architecture to realise the inference procedure of the latent variables.<n>Experiments were conducted to validate the proposed framework, including four different datasets from cardiac, brain, and abdominal medical images.
  arXiv  Detail & Related papers  (2024-01-04T08:46:39Z)
• Explainable Image Similarity: Integrating Siamese Networks and Grad-CAM [3.686808512438363]
  We propose the concept of explainable image similarity, where the goal is to provide similarity scores along with visual factual and counterfactual explanations. We present a new framework, which integrates Siamese Networks and Grad-CAM for providing explainable image similarity. The proposed approach has the potential to enhance the interpretability, trustworthiness and user acceptance of image-based systems in real-world image similarity applications.
  arXiv  Detail & Related papers  (2023-10-11T17:21:48Z)
• Attributable Visual Similarity Learning [90.69718495533144]
  This paper proposes an attributable visual similarity learning (AVSL) framework for a more accurate and explainable similarity measure between images. Motivated by the human semantic similarity cognition, we propose a generalized similarity learning paradigm to represent the similarity between two images with a graph. Experiments on the CUB-200-2011, Cars196, and Stanford Online Products datasets demonstrate significant improvements over existing deep similarity learning methods.
  arXiv  Detail & Related papers  (2022-03-28T17:35:31Z)
• Batch Curation for Unsupervised Contrastive Representation Learning [21.83249229426828]
  We introduce a textitbatch curation scheme that selects batches during the training process that are more inline with the underlying contrastive objective. We provide insights into what constitutes beneficial similar and dissimilar pairs as well as validate textitbatch curation on CIFAR10.
  arXiv  Detail & Related papers  (2021-08-19T12:14:50Z)
• Unsupervised Landmark Learning from Unpaired Data [117.81440795184587]
  Recent attempts for unsupervised landmark learning leverage synthesized image pairs that are similar in appearance but different in poses. We propose a cross-image cycle consistency framework which applies the swapping-reconstruction strategy twice to obtain the final supervision. Our proposed framework is shown to outperform strong baselines by a large margin.
  arXiv  Detail & Related papers  (2020-06-29T13:57:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.