Neural Bregman Divergences for Distance Learning

• URL: http://arxiv.org/abs/2206.04763v2
• Date: Mon, 20 Nov 2023 22:37:46 GMT
• Title: Neural Bregman Divergences for Distance Learning
• Authors: Fred Lu, Edward Raff, Francis Ferraro
• Abstract summary: We propose a new approach to learning arbitrary Bregman divergences in a differentiable manner via input convex neural networks. We show that our method more faithfully learns divergences over a set of both new and previously studied tasks. Our tests further extend to known asymmetric, but non-Bregman tasks, where our method still performs competitively despite misspecification.
• Score: 60.375385370556145
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many metric learning tasks, such as triplet learning, nearest neighbor retrieval, and visualization, are treated primarily as embedding tasks where the ultimate metric is some variant of the Euclidean distance (e.g., cosine or Mahalanobis), and the algorithm must learn to embed points into the pre-chosen space. The study of non-Euclidean geometries is often not explored, which we believe is due to a lack of tools for learning non-Euclidean measures of distance. Recent work has shown that Bregman divergences can be learned from data, opening a promising approach to learning asymmetric distances. We propose a new approach to learning arbitrary Bergman divergences in a differentiable manner via input convex neural networks and show that it overcomes significant limitations of previous works. We also demonstrate that our method more faithfully learns divergences over a set of both new and previously studied tasks, including asymmetric regression, ranking, and clustering. Our tests further extend to known asymmetric, but non-Bregman tasks, where our method still performs competitively despite misspecification, showing the general utility of our approach for asymmetric learning.

Related papers

• Disentangled Representation Learning with the Gromov-Monge Gap [65.73194652234848]
  Learning disentangled representations from unlabelled data is a fundamental challenge in machine learning. We introduce a novel approach to disentangled representation learning based on quadratic optimal transport. We demonstrate the effectiveness of our approach for quantifying disentanglement across four standard benchmarks.
  arXiv  Detail & Related papers  (2024-07-10T16:51:32Z)
• DiffKendall: A Novel Approach for Few-Shot Learning with Differentiable Kendall's Rank Correlation [16.038667928358763]
  Few-shot learning aims to adapt models trained on the base dataset to novel tasks where the categories were not seen by the model before. This often leads to a relatively uniform distribution of feature values across channels on novel classes. We show that the importance ranking of feature channels is a more reliable indicator for few-shot learning than geometric similarity metrics.
  arXiv  Detail & Related papers  (2023-07-28T05:32:56Z)
• Learning Empirical Bregman Divergence for Uncertain Distance Representation [3.9142982525021512]
  We introduce a novel method for learning empirical Bregman divergence directly from data based on parameterizing the convex function underlying the Bregman divergence with a deep learning setting. Our approach performs effectively on five popular public datasets compared to other SOTA deep metric learning methods, particularly for pattern recognition problems.
  arXiv  Detail & Related papers  (2023-04-16T04:16:28Z)
• Learning Generalized Hybrid Proximity Representation for Image Recognition [8.750658662419328]
  We propose a novel supervised metric learning method that can learn the distance metrics in both geometric and probabilistic space for image recognition. In contrast to the previous metric learning methods which usually focus on learning the distance metrics in Euclidean space, our proposed method is able to learn better distance representation in a hybrid approach.
  arXiv  Detail & Related papers  (2023-01-31T07:49:25Z)
• On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
  Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
  arXiv  Detail & Related papers  (2022-03-08T09:57:48Z)
• Provably Robust Metric Learning [98.50580215125142]
  We show that existing metric learning algorithms can result in metrics that are less robust than the Euclidean distance. We propose a novel metric learning algorithm to find a Mahalanobis distance that is robust against adversarial perturbations. Experimental results show that the proposed metric learning algorithm improves both certified robust errors and empirical robust errors.
  arXiv  Detail & Related papers  (2020-06-12T09:17:08Z)
• Towards Certified Robustness of Distance Metric Learning [53.96113074344632]
  We advocate imposing an adversarial margin in the input space so as to improve the generalization and robustness of metric learning algorithms. We show that the enlarged margin is beneficial to the generalization ability by using the theoretical technique of algorithmic robustness.
  arXiv  Detail & Related papers  (2020-06-10T16:51:53Z)
• Meta-learning with Stochastic Linear Bandits [120.43000970418939]
  We consider a class of bandit algorithms that implement a regularized version of the well-known OFUL algorithm, where the regularization is a square euclidean distance to a bias vector. We show both theoretically and experimentally, that when the number of tasks grows and the variance of the task-distribution is small, our strategies have a significant advantage over learning the tasks in isolation.
  arXiv  Detail & Related papers  (2020-05-18T08:41:39Z)
• Deep Divergence Learning [11.88774207521156]
  We introduce deep Bregman divergences, which are based on learning and parameterizing functional Bregman divergences using neural networks. We show in particular how deep metric learning formulations, kernel metric learning, Mahalanobis metric learning, and moment-matching functions for comparing distributions arise. We then describe a deep learning framework for learning general functional Bregman divergences, and show in experiments that this method yields superior performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-05-06T06:43:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.