Discriminating image representations with principal distortions

• URL: http://arxiv.org/abs/2410.15433v2
• Date: Fri, 16 May 2025 17:45:01 GMT
• Title: Discriminating image representations with principal distortions
• Authors: Jenelle Feather, David Lipshutz, Sarah E. Harvey, Alex H. Williams, Eero P. Simoncelli,
• Abstract summary: We propose a framework for comparing a set of image representations in terms of their local geometries.<n>We show how our framework can be used to probe for informative differences in local sensitivities between complex models.
• Score: 13.823252055829661
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image representations (artificial or biological) are often compared in terms of their global geometric structure; however, representations with similar global structure can have strikingly different local geometries. Here, we propose a framework for comparing a set of image representations in terms of their local geometries. We quantify the local geometry of a representation using the Fisher information matrix, a standard statistical tool for characterizing the sensitivity to local stimulus distortions, and use this as a substrate for a metric on the local geometry in the vicinity of a base image. This metric may then be used to optimally differentiate a set of models, by finding a pair of "principal distortions" that maximize the variance of the models under this metric. As an example, we use this framework to compare a set of simple models of the early visual system, identifying a novel set of image distortions that allow immediate comparison of the models by visual inspection. In a second example, we apply our method to a set of deep neural network models and reveal differences in the local geometry that arise due to architecture and training types. These examples demonstrate how our framework can be used to probe for informative differences in local sensitivities between complex models, and suggest how it could be used to compare model representations with human perception.

Related papers

• Connecting Neural Models Latent Geometries with Relative Geodesic Representations [21.71782603770616]
  We show that when a latent structure is shared between distinct latent spaces, relative distances between representations can be preserved, up to distortions.<n>We assume that distinct neural models parametrize approximately the same underlying manifold, and introduce a representation based on the pullback metric.<n>We validate our method on model stitching and retrieval tasks, covering autoencoders and vision foundation discriminative models.
  arXiv  Detail & Related papers  (2025-06-02T12:34:55Z)
• Geometry Distributions [51.4061133324376]
  We propose a novel geometric data representation that models geometry as distributions. Our approach uses diffusion models with a novel network architecture to learn surface point distributions. We evaluate our representation qualitatively and quantitatively across various object types, demonstrating its effectiveness in achieving high geometric fidelity.
  arXiv  Detail & Related papers  (2024-11-25T04:06:48Z)
• Reinforcing Pre-trained Models Using Counterfactual Images [54.26310919385808]
  This paper proposes a novel framework to reinforce classification models using language-guided generated counterfactual images. We identify model weaknesses by testing the model using the counterfactual image dataset. We employ the counterfactual images as an augmented dataset to fine-tune and reinforce the classification model.
  arXiv  Detail & Related papers  (2024-06-19T08:07:14Z)
• Intriguing Differences Between Zero-Shot and Systematic Evaluations of Vision-Language Transformer Models [7.360937524701675]
  Transformer-based models have dominated natural language processing and other areas in the last few years due to their superior (zero-shot) performance on benchmark datasets. In this paper, based on a new gradient descent optimization method, we are able to explore the embedding space of a commonly used vision-language model. Using the Imagenette dataset, we show that while the model achieves over 99% zero-shot classification performance, it fails systematic evaluations completely.
  arXiv  Detail & Related papers  (2024-02-13T14:07:49Z)
• Bayesian Unsupervised Disentanglement of Anatomy and Geometry for Deep Groupwise Image Registration [50.62725807357586]
  This article presents a general Bayesian learning framework for multi-modal groupwise image registration. We propose a novel hierarchical variational auto-encoding architecture to realise the inference procedure of the latent variables. Experiments were conducted to validate the proposed framework, including four different datasets from cardiac, brain, and abdominal medical images.
  arXiv  Detail & Related papers  (2024-01-04T08:46:39Z)
• Curved Diffusion: A Generative Model With Optical Geometry Control [56.24220665691974]
  The influence of different optical systems on the final scene appearance is frequently overlooked. This study introduces a framework that intimately integrates a textto-image diffusion model with the particular lens used in image rendering.
  arXiv  Detail & Related papers  (2023-11-29T13:06:48Z)
• Comparing Foundation Models using Data Kernels [13.099029073152257]
  We present a methodology for directly comparing the embedding space geometry of foundation models. Our methodology is grounded in random graph theory and enables valid hypothesis testing of embedding similarity. We show how our framework can induce a manifold of models equipped with a distance function that correlates strongly with several downstream metrics.
  arXiv  Detail & Related papers  (2023-05-09T02:01:07Z)
• Improving Shape Awareness and Interpretability in Deep Networks Using Geometric Moments [0.0]
  Deep networks for image classification often rely more on texture information than object shape. This paper presents a deep-learning model inspired by geometric moments. We demonstrate the effectiveness of our method on standard image classification datasets.
  arXiv  Detail & Related papers  (2022-05-24T02:08:05Z)
• An application of a pseudo-parabolic modeling to texture image recognition [0.0]
  We present a novel methodology for texture image recognition using a partial differential equation modeling. We employ the pseudo-parabolic Buckley-Leverett equation to provide a dynamics to the digital image representation and collect local descriptors from those images evolving in time.
  arXiv  Detail & Related papers  (2021-02-09T18:08:42Z)
• SIR: Self-supervised Image Rectification via Seeing the Same Scene from Multiple Different Lenses [82.56853587380168]
  We propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted images of the same scene from different lens should be the same. We leverage a differentiable warping module to generate the rectified images and re-distorted images from the distortion parameters. Our method achieves comparable or even better performance than the supervised baseline method and representative state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-11-30T08:23:25Z)
• Reorganizing local image features with chaotic maps: an application to texture recognition [0.0]
  We propose a chaos-based local descriptor for texture recognition. We map the image into the three-dimensional Euclidean space, iterate a chaotic map over this three-dimensional structure and convert it back to the original image. The performance of our method was verified on the classification of benchmark databases and in the identification of Brazilian plant species based on the texture of the leaf surface.
  arXiv  Detail & Related papers  (2020-07-15T03:15:01Z)
• Geometrically Mappable Image Features [85.81073893916414]
  Vision-based localization of an agent in a map is an important problem in robotics and computer vision. We propose a method that learns image features targeted for image-retrieval-based localization.
  arXiv  Detail & Related papers  (2020-03-21T15:36:38Z)
• Multi-View Optimization of Local Feature Geometry [70.18863787469805]
  We address the problem of refining the geometry of local image features from multiple views without known scene or camera geometry. Our proposed method naturally complements the traditional feature extraction and matching paradigm. We show that our method consistently improves the triangulation and camera localization performance for both hand-crafted and learned local features.
  arXiv  Detail & Related papers  (2020-03-18T17:22:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.