Image Similarity using An Ensemble of Context-Sensitive Models

• URL: http://arxiv.org/abs/2401.07951v2
• Date: Tue, 10 Sep 2024 13:33:37 GMT
• Title: Image Similarity using An Ensemble of Context-Sensitive Models
• Authors: Zukang Liao, Min Chen,
• Abstract summary: We present a more intuitive approach to build and compare image similarity models based on labelled data. We address the challenges of sparse sampling in the image space (R, A, B) and biases in the models trained with context-based data. Our testing results show that the ensemble model constructed performs 5% better than the best individual context-sensitive models.
• Score: 2.9490616593440317
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image similarity has been extensively studied in computer vision. In recent years, machine-learned models have shown their ability to encode more semantics than traditional multivariate metrics. However, in labelling semantic similarity, assigning a numerical score to a pair of images is impractical, making the improvement and comparisons on the task difficult. In this work, we present a more intuitive approach to build and compare image similarity models based on labelled data in the form of A:R vs B:R, i.e., determining if an image A is closer to a reference image R than another image B. We address the challenges of sparse sampling in the image space (R, A, B) and biases in the models trained with context-based data by using an ensemble model. Our testing results show that the ensemble model constructed performs ~5% better than the best individual context-sensitive models. They also performed better than the models that were directly fine-tuned using mixed imagery data as well as existing deep embeddings, e.g., CLIP and DINO. This work demonstrates that context-based labelling and model training can be effective when an appropriate ensemble approach is used to alleviate the limitation due to sparse sampling.

Related papers

• Fast constrained sampling in pre-trained diffusion models [77.21486516041391]
  Diffusion models have dominated the field of large, generative image models. We propose an algorithm for fast-constrained sampling in large pre-trained diffusion models.
  arXiv  Detail & Related papers  (2024-10-24T14:52:38Z)
• Reinforcing Pre-trained Models Using Counterfactual Images [54.26310919385808]
  This paper proposes a novel framework to reinforce classification models using language-guided generated counterfactual images. We identify model weaknesses by testing the model using the counterfactual image dataset. We employ the counterfactual images as an augmented dataset to fine-tune and reinforce the classification model.
  arXiv  Detail & Related papers  (2024-06-19T08:07:14Z)
• CorrEmbed: Evaluating Pre-trained Model Image Similarity Efficacy with a Novel Metric [6.904776368895614]
  We evaluate the viability of the image embeddings from pre-trained computer vision models using a novel approach named CorrEmbed. Our approach computes the correlation between distances in image embeddings and distances in human-generated tag vectors. Our method also identifies deviations from this pattern, providing insights into how different models capture high-level image features.
  arXiv  Detail & Related papers  (2023-08-30T16:23:07Z)
• Evaluating Data Attribution for Text-to-Image Models [62.844382063780365]
  We evaluate attribution through "customization" methods, which tune an existing large-scale model toward a given exemplar object or style. Our key insight is that this allows us to efficiently create synthetic images that are computationally influenced by the exemplar by construction. By taking into account the inherent uncertainty of the problem, we can assign soft attribution scores over a set of training images.
  arXiv  Detail & Related papers  (2023-06-15T17:59:51Z)
• Masked Images Are Counterfactual Samples for Robust Fine-tuning [77.82348472169335]
  Fine-tuning deep learning models can lead to a trade-off between in-distribution (ID) performance and out-of-distribution (OOD) robustness. We propose a novel fine-tuning method, which uses masked images as counterfactual samples that help improve the robustness of the fine-tuning model.
  arXiv  Detail & Related papers  (2023-03-06T11:51:28Z)
• Effective Robustness against Natural Distribution Shifts for Models with Different Training Data [113.21868839569]
  "Effective robustness" measures the extra out-of-distribution robustness beyond what can be predicted from the in-distribution (ID) performance. We propose a new evaluation metric to evaluate and compare the effective robustness of models trained on different data.
  arXiv  Detail & Related papers  (2023-02-02T19:28:41Z)
• Through a fair looking-glass: mitigating bias in image datasets [1.0323063834827415]
  We present a fast and effective model to de-bias an image dataset through reconstruction and minimizing the statistical dependence between intended variables. We evaluate our proposed model on CelebA dataset, compare the results with a state-of-the-art de-biasing method, and show that the model achieves a promising fairness-accuracy combination.
  arXiv  Detail & Related papers  (2022-09-18T20:28:36Z)
• Identical Image Retrieval using Deep Learning [0.0]
  We are using the BigTransfer Model, which is a state-of-art model itself. We extract the key features and train on the K-Nearest Neighbor model to obtain the nearest neighbor. The application of our model is to find similar images, which are hard to achieve through text queries within a low inference time.
  arXiv  Detail & Related papers  (2022-05-10T13:34:41Z)
• IMACS: Image Model Attribution Comparison Summaries [16.80986701058596]
  We introduce IMACS, a method that combines gradient-based model attributions with aggregation and visualization techniques. IMACS extracts salient input features from an evaluation dataset, clusters them based on similarity, then visualizes differences in model attributions for similar input features. We show how our technique can uncover behavioral differences caused by domain shift between two models trained on satellite images.
  arXiv  Detail & Related papers  (2022-01-26T21:35:14Z)
• Evaluating Contrastive Models for Instance-based Image Retrieval [6.393147386784114]
  We evaluate contrastive models for the task of image retrieval. We find that models trained using contrastive methods perform on-par with (and outperforms) a pre-trained baseline trained on the ImageNet labels.
  arXiv  Detail & Related papers  (2021-04-30T12:05:23Z)
• I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively [135.7695909882746]
  We name the MAximum Discrepancy (MAD) competition. We adaptively sample a small test set from an arbitrarily large corpus of unlabeled images. Human labeling on the resulting model-dependent image sets reveals the relative performance of the competing classifiers.
  arXiv  Detail & Related papers  (2020-02-25T03:32:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.