Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning

• URL: http://arxiv.org/abs/2107.09562v1
• Date: Tue, 20 Jul 2021 15:26:09 GMT
• Title: Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning
• Authors: Timo Milbich, Karsten Roth, Samarth Sinha, Ludwig Schmidt, Marzyeh Ghassemi, Bj\"orn Ommer
• Abstract summary: We present the ooDML benchmark to characterize generalization under out-of-distribution shifts in DML. ooDML is designed to probe the generalization performance on much more challenging, diverse train-to-test distribution shifts. We find that while generalization tends to consistently degrade with difficulty, some methods are better at retaining performance as the distribution shift increases.
• Score: 32.51394862932118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep Metric Learning (DML) aims to find representations suitable for zero-shot transfer to a priori unknown test distributions. However, common evaluation protocols only test a single, fixed data split in which train and test classes are assigned randomly. More realistic evaluations should consider a broad spectrum of distribution shifts with potentially varying degree and difficulty. In this work, we systematically construct train-test splits of increasing difficulty and present the ooDML benchmark to characterize generalization under out-of-distribution shifts in DML. ooDML is designed to probe the generalization performance on much more challenging, diverse train-to-test distribution shifts. Based on our new benchmark, we conduct a thorough empirical analysis of state-of-the-art DML methods. We find that while generalization tends to consistently degrade with difficulty, some methods are better at retaining performance as the distribution shift increases. Finally, we propose few-shot DML as an efficient way to consistently improve generalization in response to unknown test shifts presented in ooDML. Code available here: https://github.com/Confusezius/Characterizing_Generalization_in_DeepMetricLearning.

Related papers

• Understanding the Generalizability of Link Predictors Under Distribution Shifts on Graphs [34.58496513149175]
  Many popular benchmark datasets assume that dataset samples are drawn from the same distribution. We introduce LP-specific data splits which utilize structural properties to induce a controlled distribution shift. We verify the shift's effect empirically through evaluation of different SOTA LP methods and subsequently couple these methods with generalization techniques.
  arXiv  Detail & Related papers  (2024-06-13T03:47:12Z)
• Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition [114.96385572118042]
  We argue that the variation in test label distributions can be broken down hierarchically into global and local levels. We propose a new MoE strategy, $mathsfDirMixE$, which assigns experts to different Dirichlet meta-distributions of the label distribution. We show that our proposed objective benefits from enhanced generalization by virtue of the variance-based regularization.
  arXiv  Detail & Related papers  (2024-05-13T14:24:56Z)
• Enhancing Out-of-Distribution Detection with Multitesting-based Layer-wise Feature Fusion [11.689517005768046]
  Out-of-distribution samples may exhibit shifts in local or global features compared to the training distribution. We propose a novel framework, Multitesting-based Layer-wise Out-of-Distribution (OOD) Detection. Our scheme effectively enhances the performance of out-of-distribution detection when compared to baseline methods.
  arXiv  Detail & Related papers  (2024-03-16T04:35:04Z)
• Invariant Anomaly Detection under Distribution Shifts: A Causal Perspective [6.845698872290768]
  Anomaly detection (AD) is the machine learning task of identifying highly discrepant abnormal samples. Under the constraints of a distribution shift, the assumption that training samples and test samples are drawn from the same distribution breaks down. We attempt to increase the resilience of anomaly detection models to different kinds of distribution shifts.
  arXiv  Detail & Related papers  (2023-12-21T23:20:47Z)
• Learning Invariant Molecular Representation in Latent Discrete Space [52.13724532622099]
  We propose a new framework for learning molecular representations that exhibit invariance and robustness against distribution shifts. Our model achieves stronger generalization against state-of-the-art baselines in the presence of various distribution shifts.
  arXiv  Detail & Related papers  (2023-10-22T04:06:44Z)
• Distribution Shift Inversion for Out-of-Distribution Prediction [57.22301285120695]
  We propose a portable Distribution Shift Inversion algorithm for Out-of-Distribution (OoD) prediction. We show that our method provides a general performance gain when plugged into a wide range of commonly used OoD algorithms.
  arXiv  Detail & Related papers  (2023-06-14T08:00:49Z)
• Anomaly Detection under Distribution Shift [24.094884041252044]
  Anomaly detection (AD) is a crucial machine learning task that aims to learn patterns from a set of normal training samples to identify abnormal samples in test data. Most existing AD studies assume that the training and test data are drawn from the same data distribution, but the test data can have large distribution shifts. We introduce a novel robust AD approach to diverse distribution shifts by minimizing the distribution gap between in-distribution and OOD normal samples in both the training and inference stages.
  arXiv  Detail & Related papers  (2023-03-24T07:39:08Z)
• Meta-Causal Feature Learning for Out-of-Distribution Generalization [71.38239243414091]
  This paper presents a balanced meta-causal learner (BMCL), which includes a balanced task generation module (BTG) and a meta-causal feature learning module (MCFL) BMCL effectively identifies the class-invariant visual regions for classification and may serve as a general framework to improve the performance of the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-08-22T09:07:02Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• GANs with Variational Entropy Regularizers: Applications in Mitigating the Mode-Collapse Issue [95.23775347605923]
  Building on the success of deep learning, Generative Adversarial Networks (GANs) provide a modern approach to learn a probability distribution from observed samples. GANs often suffer from the mode collapse issue where the generator fails to capture all existing modes of the input distribution. We take an information-theoretic approach and maximize a variational lower bound on the entropy of the generated samples to increase their diversity.
  arXiv  Detail & Related papers  (2020-09-24T19:34:37Z)
• Revisiting Training Strategies and Generalization Performance in Deep Metric Learning [28.54755295856929]
  We revisit the most widely used DML objective functions and conduct a study of the crucial parameter choices. Under consistent comparison, DML objectives show much higher saturation than indicated by literature. Exploiting these insights, we propose a simple, yet effective, training regularization to reliably boost the performance of ranking-based DML models.
  arXiv  Detail & Related papers  (2020-02-19T22:16:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.