Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information Visualization

• URL: http://arxiv.org/abs/2503.00228v1
• Date: Fri, 28 Feb 2025 22:25:04 GMT
• Title: Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information Visualization
• Authors: Sheng Long, Angelos Chatzimparmpas, Emma Alexander, Matthew Kay, Jessica Hullman,
• Abstract summary: Recent studies show deep-feature-based similarity metrics correlate well with perceptual judgments of image similarity.<n>We extend a similarity metric using five ML architectures and three pre-trained weight sets.<n>Our work contributes to understanding how deep-feature-based metrics can enhance similarity assessments in visualization, potentially improving visual analysis tools and techniques.
• Score: 22.920857549451668
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Judging the similarity of visualizations is crucial to various applications, such as visualization-based search and visualization recommendation systems. Recent studies show deep-feature-based similarity metrics correlate well with perceptual judgments of image similarity and serve as effective loss functions for tasks like image super-resolution and style transfer. We explore the application of such metrics to judgments of visualization similarity. We extend a similarity metric using five ML architectures and three pre-trained weight sets. We replicate results from previous crowd-sourced studies on scatterplot and visual channel similarity perception. Notably, our metric using pre-trained ImageNet weights outperformed gradient-descent tuned MS-SSIM, a multi-scale similarity metric based on luminance, contrast, and structure. Our work contributes to understanding how deep-feature-based metrics can enhance similarity assessments in visualization, potentially improving visual analysis tools and techniques. Supplementary materials are available at https://osf.io/dj2ms.

Related papers

• DiffSim: Taming Diffusion Models for Evaluating Visual Similarity [19.989551230170584]
  This paper introduces the DiffSim method to measure visual similarity in generative models.<n>By aligning features in the attention layers of the denoising U-Net, DiffSim evaluates both appearance and style similarity.<n>We also introduce the Sref and IP benchmarks to evaluate visual similarity at the level of style and instance.
  arXiv  Detail & Related papers  (2024-12-19T07:00:03Z)
• Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling [58.50618448027103]
  Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning.<n>This paper explores the differences across various CLIP-trained vision backbones.<n>Method achieves a remarkable increase in accuracy of up to 39.1% over the best single backbone.
  arXiv  Detail & Related papers  (2024-05-27T12:59:35Z)
• Learning an Adaptation Function to Assess Image Visual Similarities [0.0]
  We focus here on the specific task of learning visual image similarities when analogy matters. We propose to compare different supervised, semi-supervised and self-supervised networks, pre-trained on distinct scales and contents datasets. Our experiments conducted on the Totally Looks Like image dataset highlight the interest of our method, by increasing the retrieval scores of the best model @1 by 2.25x.
  arXiv  Detail & Related papers  (2022-06-03T07:15:00Z)
• Attributable Visual Similarity Learning [90.69718495533144]
  This paper proposes an attributable visual similarity learning (AVSL) framework for a more accurate and explainable similarity measure between images. Motivated by the human semantic similarity cognition, we propose a generalized similarity learning paradigm to represent the similarity between two images with a graph. Experiments on the CUB-200-2011, Cars196, and Stanford Online Products datasets demonstrate significant improvements over existing deep similarity learning methods.
  arXiv  Detail & Related papers  (2022-03-28T17:35:31Z)
• A study of deep perceptual metrics for image quality assessment [3.254879465902239]
  We study perceptual metrics based on deep neural networks for tackling the Image Quality Assessment (IQA) task. We propose our multi-resolution perceptual metric (MR-Perceptual) that allows us to aggregate perceptual information at different resolutions.
  arXiv  Detail & Related papers  (2022-02-17T14:52:53Z)
• Contextual Similarity Aggregation with Self-attention for Visual Re-ranking [96.55393026011811]
  We propose a visual re-ranking method by contextual similarity aggregation with self-attention. We conduct comprehensive experiments on four benchmark datasets to demonstrate the generality and effectiveness of our proposed visual re-ranking method.
  arXiv  Detail & Related papers  (2021-10-26T06:20:31Z)
• Deep Relational Metric Learning [84.95793654872399]
  This paper presents a deep relational metric learning framework for image clustering and retrieval. We learn an ensemble of features that characterizes an image from different aspects to model both interclass and intraclass distributions. Experiments on the widely-used CUB-200-2011, Cars196, and Stanford Online Products datasets demonstrate that our framework improves existing deep metric learning methods and achieves very competitive results.
  arXiv  Detail & Related papers  (2021-08-23T09:31:18Z)
• Towards Interpretable Deep Metric Learning with Structural Matching [86.16700459215383]
  We present a deep interpretable metric learning (DIML) method for more transparent embedding learning. Our method is model-agnostic, which can be applied to off-the-shelf backbone networks and metric learning methods. We evaluate our method on three major benchmarks of deep metric learning including CUB200-2011, Cars196, and Stanford Online Products.
  arXiv  Detail & Related papers  (2021-08-12T17:59:09Z)
• Determining Image similarity with Quasi-Euclidean Metric [0.0]
  We evaluate Quasi-Euclidean metric as an image similarity measure and analyze how it fares against the existing standard ways like SSIM and Euclidean metric. In some cases, our methodology projected remarkable performance and it is also interesting to note that our implementation proves to be a step ahead in recognizing similarity.
  arXiv  Detail & Related papers  (2020-06-25T18:12:21Z)
• Geometrically Mappable Image Features [85.81073893916414]
  Vision-based localization of an agent in a map is an important problem in robotics and computer vision. We propose a method that learns image features targeted for image-retrieval-based localization.
  arXiv  Detail & Related papers  (2020-03-21T15:36:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.