Identifying and Mitigating Flaws of Deep Perceptual Similarity Metrics

• URL: http://arxiv.org/abs/2207.02512v1
• Date: Wed, 6 Jul 2022 08:28:39 GMT
• Title: Identifying and Mitigating Flaws of Deep Perceptual Similarity Metrics
• Authors: Oskar Sj\"ogren, Gustav Grund Pihlgren, Fredrik Sandin, Marcus Liwicki
• Abstract summary: This work investigates the benefits and flaws of the Deep Perceptual Similarity (DPS) metric. The metrics are analyzed in-depth to understand the strengths and weaknesses of the metrics. This work contributes with new insights into the flaws of DPS, and further suggests improvements to the metrics.
• Score: 1.484528358552186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Measuring the similarity of images is a fundamental problem to computer vision for which no universal solution exists. While simple metrics such as the pixel-wise L2-norm have been shown to have significant flaws, they remain popular. One group of recent state-of-the-art metrics that mitigates some of those flaws are Deep Perceptual Similarity (DPS) metrics, where the similarity is evaluated as the distance in the deep features of neural networks. However, DPS metrics themselves have been less thoroughly examined for their benefits and, especially, their flaws. This work investigates the most common DPS metric, where deep features are compared by spatial position, along with metrics comparing the averaged and sorted deep features. The metrics are analyzed in-depth to understand the strengths and weaknesses of the metrics by using images designed specifically to challenge them. This work contributes with new insights into the flaws of DPS, and further suggests improvements to the metrics. An implementation of this work is available online: https://github.com/guspih/deep_perceptual_similarity_analysis/

Related papers

• Size-invariance Matters: Rethinking Metrics and Losses for Imbalanced Multi-object Salient Object Detection [133.66006666465447]
  Current metrics are size-sensitive, where larger objects are focused, and smaller ones tend to be ignored. We argue that the evaluation should be size-invariant because bias based on size is unjustified without additional semantic information. We develop an optimization framework tailored to this goal, achieving considerable improvements in detecting objects of different sizes.
  arXiv  Detail & Related papers  (2024-05-16T03:01:06Z)
• UniDepth: Universal Monocular Metric Depth Estimation [81.80512457953903]
  We propose a new model, UniDepth, capable of reconstructing metric 3D scenes from solely single images across domains. Our model exploits a pseudo-spherical output representation, which disentangles camera and depth representations. Thorough evaluations on ten datasets in a zero-shot regime consistently demonstrate the superior performance of UniDepth.
  arXiv  Detail & Related papers  (2024-03-27T18:06:31Z)
• LipSim: A Provably Robust Perceptual Similarity Metric [56.03417732498859]
  We show the vulnerability of state-of-the-art perceptual similarity metrics based on an ensemble of ViT-based feature extractors to adversarial attacks. We then propose a framework to train a robust perceptual similarity metric called LipSim with provable guarantees. LipSim provides guarded areas around each data point and certificates for all perturbations within an $ell$ ball.
  arXiv  Detail & Related papers  (2023-10-27T16:59:51Z)
• A Geometrical Approach to Evaluate the Adversarial Robustness of Deep Neural Networks [52.09243852066406]
  Adversarial Converging Time Score (ACTS) measures the converging time as an adversarial robustness metric. We validate the effectiveness and generalization of the proposed ACTS metric against different adversarial attacks on the large-scale ImageNet dataset.
  arXiv  Detail & Related papers  (2023-10-10T09:39:38Z)
• R-LPIPS: An Adversarially Robust Perceptual Similarity Metric [71.33812578529006]
  We propose the Robust Learned Perceptual Image Patch Similarity (R-LPIPS) metric. R-LPIPS is a new metric that leverages adversarially trained deep features. We demonstrate the superiority of R-LPIPS compared to the classical LPIPS metric.
  arXiv  Detail & Related papers  (2023-07-27T19:11:31Z)
• Deep Perceptual Similarity is Adaptable to Ambiguous Contexts [1.6217405839281338]
  The concept of image similarity is ambiguous, and images can be similar in one context and not in another. This work explores the ability of deep perceptual similarity (DPS) metrics to adapt to a given context. The adapted metrics are evaluated on a perceptual similarity dataset to evaluate if adapting to a ranking affects their prior performance.
  arXiv  Detail & Related papers  (2023-04-05T07:31:44Z)
• Evaluation of FEM and MLFEM AI-explainers in Image Classification tasks with reference-based and no-reference metrics [0.0]
  We remind recently proposed post-hoc explainers FEM and MLFEM which have been designed for explanations of CNNs in image and video classification tasks. We propose their evaluation with reference-based and no-reference metrics. As a no-reference metric we use "stability" metric, proposed by Alvarez-Melis and Jaakkola.
  arXiv  Detail & Related papers  (2022-12-02T14:55:31Z)
• Shift-tolerant Perceptual Similarity Metric [5.326626090397465]
  Existing perceptual similarity metrics assume an image and its reference are well aligned. This paper studies the effect of small misalignment, specifically a small shift between the input and reference image, on existing metrics. We develop a new deep neural network-based perceptual similarity metric.
  arXiv  Detail & Related papers  (2022-07-27T17:55:04Z)
• Towards Interpretable Deep Metric Learning with Structural Matching [86.16700459215383]
  We present a deep interpretable metric learning (DIML) method for more transparent embedding learning. Our method is model-agnostic, which can be applied to off-the-shelf backbone networks and metric learning methods. We evaluate our method on three major benchmarks of deep metric learning including CUB200-2011, Cars196, and Stanford Online Products.
  arXiv  Detail & Related papers  (2021-08-12T17:59:09Z)
• Reliable Fidelity and Diversity Metrics for Generative Models [30.941563781926202]
  The most widely used metric for measuring the similarity between real and generated images has been the Fr'echet Inception Distance (FID) score. We show that even the latest version of the precision and recall metrics are not reliable yet. We propose density and coverage metrics that solve the above issues.
  arXiv  Detail & Related papers  (2020-02-23T00:50:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.