The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

• URL: http://arxiv.org/abs/2607.18237v1
• Date: Mon, 20 Jul 2026 17:59:51 GMT
• Title: The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric
• Abstract summary: We introduce a large-scale dataset of human similarity judgments over image triplets.<n>We fine-tune a vision-language model to produce our Text-Prompted Image Perceptual Similarity metric.<n>TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution.
• Score: 78.74729751232852
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Human visual similarity judgments are context-dependent. For example, two images may be similar in shape but distinct in color. Existing perceptual similarity metrics, however, collapse these nuances into a single scalar value, offering no mechanism to condition on specific aspects. To bridge this gap, we introduce a large-scale dataset of human similarity judgments over image triplets, where each triplet is annotated across multiple, free-form semantic aspects of similarity. Benchmarking a broad range of frontier vision-language models (VLMs) reveals a considerable performance gap compared to human annotators' consensus. Leveraging our data, we fine-tune a VLM to produce our Text-Prompted Image Perceptual Similarity (TPIPS) metric, capturing multiple senses of visual similarity depending on the specified text prompt. We demonstrate that TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution. Finally, we show that TPIPS unlocks new capabilities in text-guided retrieval, compositional search, and the fine-grained evaluation of generative models. Our code, data, and trained models are at https://peterwang512.github.io/TPIPS

Related papers

• Generative Retrieval for Unsupervised Text-Based Person Search [77.46429603553388]
  Text-based person search (TBPS) aims to retrieve images of a target person from a large image gallery based on a given natural language description.<n>We propose GTR+, a two-stage generation-then-retrieval framework.<n>We also contribute LargeFine-Person, a large-scale TBPS dataset with high-quality, fine-grained, and diverse textual annotations.
  arXiv  Detail & Related papers  (2026-09-11T15:25:00Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Discovering Divergent Representations between Text-to-Image Models [87.40710629963264]
  We investigate when and how visual representations learned by two different generative models diverge.<n>We introduce CompCon, an evolutionary search algorithm that discovers visual attributes more prevalent in one model's output than the other.<n>We use CompCon to compare popular text-to-image models, finding divergent representations such as how PixArt depicts prompts mentioning loneliness with wet streets.
  arXiv  Detail & Related papers  (2025-09-10T19:07:55Z)
• Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment [25.209622555403527]
  We propose a framework called Asymmetric Visual Semantic Embedding (AVSE) to dynamically select features from various regions of images tailored to different textual inputs for similarity calculation.<n>AVSE calculates visual semantic similarity by finding the optimal match of meta-semantic embeddings of two modalities.<n>Our proposed AVSE model is extensively evaluated on the large-scale MS-COCO and Flickr30K datasets.
  arXiv  Detail & Related papers  (2025-03-10T06:38:41Z)
• Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding [112.0878081944858]
  Quantifying the degree of similarity between images is a key copyright issue for image-based machine learning. We seek to define and compute a notion of "conceptual similarity" among images that captures high-level relations. Two highly dissimilar images can be discriminated early in their description, whereas conceptually dissimilar ones will need more detail to be distinguished.
  arXiv  Detail & Related papers  (2024-02-14T03:31:17Z)
• DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data [43.247597420676044]
  Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. We develop a perceptual metric that assesses images holistically.
  arXiv  Detail & Related papers  (2023-06-15T17:59:50Z)
• Deep Perceptual Similarity is Adaptable to Ambiguous Contexts [1.6217405839281338]
  The concept of image similarity is ambiguous, and images can be similar in one context and not in another. This work explores the ability of deep perceptual similarity (DPS) metrics to adapt to a given context. The adapted metrics are evaluated on a perceptual similarity dataset to evaluate if adapting to a ranking affects their prior performance.
  arXiv  Detail & Related papers  (2023-04-05T07:31:44Z)
• Learning an Adaptation Function to Assess Image Visual Similarities [0.0]
  We focus here on the specific task of learning visual image similarities when analogy matters. We propose to compare different supervised, semi-supervised and self-supervised networks, pre-trained on distinct scales and contents datasets. Our experiments conducted on the Totally Looks Like image dataset highlight the interest of our method, by increasing the retrieval scores of the best model @1 by 2.25x.
  arXiv  Detail & Related papers  (2022-06-03T07:15:00Z)
• Attributable Visual Similarity Learning [90.69718495533144]
  This paper proposes an attributable visual similarity learning (AVSL) framework for a more accurate and explainable similarity measure between images. Motivated by the human semantic similarity cognition, we propose a generalized similarity learning paradigm to represent the similarity between two images with a graph. Experiments on the CUB-200-2011, Cars196, and Stanford Online Products datasets demonstrate significant improvements over existing deep similarity learning methods.
  arXiv  Detail & Related papers  (2022-03-28T17:35:31Z)
• Learning Contrastive Representation for Semantic Correspondence [150.29135856909477]
  We propose a multi-level contrastive learning approach for semantic matching. We show that image-level contrastive learning is a key component to encourage the convolutional features to find correspondence between similar objects.
  arXiv  Detail & Related papers  (2021-09-22T18:34:14Z)
• Deep Multimodal Image-Text Embeddings for Automatic Cross-Media Retrieval [0.0]
  We introduce an end-to-end deep multimodal convolutional-recurrent network for learning both vision and language representations simultaneously. The model learns which pairs are a match (positive) and which ones are a mismatch (negative) using a hinge-based triplet ranking.
  arXiv  Detail & Related papers  (2020-02-23T23:58:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.