ELM: Embedding and Logit Margins for Long-Tail Learning

• URL: http://arxiv.org/abs/2204.13208v1
• Date: Wed, 27 Apr 2022 21:53:50 GMT
• Title: ELM: Embedding and Logit Margins for Long-Tail Learning
• Authors: Wittawat Jitkrittum, Aditya Krishna Menon, Ankit Singh Rawat, Sanjiv Kumar
• Abstract summary: Long-tail learning is the problem of learning under skewed label distributions. We present Embedding and Logit Margins (ELM), a unified approach to enforce margins in logit space. The ELM method is shown to perform well empirically, and results in tighter tail class embeddings.
• Score: 70.19006872113862
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-tail learning is the problem of learning under skewed label distributions, which pose a challenge for standard learners. Several recent approaches for the problem have proposed enforcing a suitable margin in logit space. Such techniques are intuitive analogues of the guiding principle behind SVMs, and are equally applicable to linear models and neural models. However, when applied to neural models, such techniques do not explicitly control the geometry of the learned embeddings. This can be potentially sub-optimal, since embeddings for tail classes may be diffuse, resulting in poor generalization for these classes. We present Embedding and Logit Margins (ELM), a unified approach to enforce margins in logit space, and regularize the distribution of embeddings. This connects losses for long-tail learning to proposals in the literature on metric embedding, and contrastive learning. We theoretically show that minimising the proposed ELM objective helps reduce the generalisation gap. The ELM method is shown to perform well empirically, and results in tighter tail class embeddings.

Related papers

• Combating Representation Learning Disparity with Geometric Harmonization [50.29859682439571]
  We propose a novel Geometric Harmonization (GH) method to encourage category-level uniformity in representation learning. Our proposal does not alter the setting of SSL and can be easily integrated into existing methods in a low-cost manner.
  arXiv  Detail & Related papers  (2023-10-26T17:41:11Z)
• Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning [70.52689048213398]
  This paper studies the performance of overfitted meta-learning under a linear regression model with Gaussian features. We find new and interesting properties that do not exist in single-task linear regression. Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large.
  arXiv  Detail & Related papers  (2023-04-09T20:36:13Z)
• Learning Imbalanced Datasets with Maximum Margin Loss [21.305656747991026]
  A learning algorithm referred to as Maximum Margin (MM) is proposed for considering the class-imbalance data learning issue. We design a new Maximum Margin (MM) loss function, motivated by minimizing a margin-based generalization bound through the shifting decision bound.
  arXiv  Detail & Related papers  (2022-06-11T00:21:41Z)
• Guided Deep Metric Learning [0.9786690381850356]
  We propose a novel approach to DML that we call Guided Deep Metric Learning. The proposed method is capable of a better manifold generalization and representation to up to 40% improvement.
  arXiv  Detail & Related papers  (2022-06-04T17:34:11Z)
• Semi-Supervised Metric Learning: A Deep Resurrection [22.918651280720855]
  Semi-Supervised DML (SSDML) tries to learn a metric using a few labeled examples, and abundantly available unlabeled examples. We propose a graph-based approach that first propagates the affinities between the pairs of examples. We impose Metricity constraint on the metric parameters, as it leads to a better performance.
  arXiv  Detail & Related papers  (2021-05-10T12:28:45Z)
• Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning [80.20302993614594]
  We provide a statistical analysis to overcome drawbacks of Laplacian regularization. We unveil a large body of spectral filtering methods that exhibit desirable behaviors. We provide realistic computational guidelines in order to make our method usable with large amounts of data.
  arXiv  Detail & Related papers  (2020-09-09T14:28:54Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)
• Boosting Few-Shot Learning With Adaptive Margin Loss [109.03665126222619]
  This paper proposes an adaptive margin principle to improve the generalization ability of metric-based meta-learning approaches for few-shot learning problems. Extensive experiments demonstrate that the proposed method can boost the performance of current metric-based meta-learning approaches.
  arXiv  Detail & Related papers  (2020-05-28T07:58:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.