Revisiting Logistic-softmax Likelihood in Bayesian Meta-Learning for Few-Shot Classification

• URL: http://arxiv.org/abs/2310.10379v2
• Date: Thu, 10 Oct 2024 21:32:36 GMT
• Title: Revisiting Logistic-softmax Likelihood in Bayesian Meta-Learning for Few-Shot Classification
• Authors: Tianjun Ke, Haoqun Cao, Zenan Ling, Feng Zhou,
• Abstract summary: logistic-softmax is often employed as an alternative to the softmax likelihood in multi-class Gaussian process classification. We revisit and redesign the logistic-softmax likelihood, which enables control of the textita priori confidence level through a temperature parameter. Our approach yields well-calibrated uncertainty estimates and achieves comparable or superior results on standard benchmark datasets.
• Score: 4.813254903898101
• License:
• Abstract: Meta-learning has demonstrated promising results in few-shot classification (FSC) by learning to solve new problems using prior knowledge. Bayesian methods are effective at characterizing uncertainty in FSC, which is crucial in high-risk fields. In this context, the logistic-softmax likelihood is often employed as an alternative to the softmax likelihood in multi-class Gaussian process classification due to its conditional conjugacy property. However, the theoretical property of logistic-softmax is not clear and previous research indicated that the inherent uncertainty of logistic-softmax leads to suboptimal performance. To mitigate these issues, we revisit and redesign the logistic-softmax likelihood, which enables control of the \textit{a priori} confidence level through a temperature parameter. Furthermore, we theoretically and empirically show that softmax can be viewed as a special case of logistic-softmax and logistic-softmax induces a larger family of data distribution than softmax. Utilizing modified logistic-softmax, we integrate the data augmentation technique into the deep kernel based Gaussian process meta-learning framework, and derive an analytical mean-field approximation for task-specific updates. Our approach yields well-calibrated uncertainty estimates and achieves comparable or superior results on standard benchmark datasets. Code is publicly available at \url{https://github.com/keanson/revisit-logistic-softmax}.

Related papers

• r-softmax: Generalized Softmax with Controllable Sparsity Rate [11.39524236962986]
  We propose r-softmax, a modification of the softmax, outputting sparse probability distribution with controllable sparsity rate. We show on several multi-label datasets that r-softmax outperforms other sparse alternatives to softmax and is highly competitive with the original softmax.
  arXiv  Detail & Related papers  (2023-04-11T14:28:29Z)
• Revisiting Softmax for Uncertainty Approximation in Text Classification [45.07154956156555]
  Uncertainty approximation in text classification is an important area with applications in domain adaptation and interpretability. One of the most widely used uncertainty approximation methods is Monte Carlo (MC) Dropout, which is computationally expensive. We compare softmax and an efficient version of MC Dropout on their uncertainty approximations and downstream text classification performance. We find that, while MC dropout produces the best uncertainty approximations, using a simple softmax leads to competitive and in some cases better uncertainty estimation for text classification at a much lower computational cost.
  arXiv  Detail & Related papers  (2022-10-25T14:13:53Z)
• Enhancing Classifier Conservativeness and Robustness by Polynomiality [23.099278014212146]
  We show howconditionality can remedy the situation. A directly related, simple, yet important technical novelty we subsequently present is softRmax. We show that two aspects of softRmax, conservativeness and inherent robustness, lead to adversarial regularization.
  arXiv  Detail & Related papers  (2022-03-23T19:36:19Z)
• Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation [101.22379613810881]
  We consider data-driven optimization problems where one must maximize a function given only queries at a fixed set of points. This problem setting emerges in many domains where function evaluation is a complex and expensive process. We propose a tractable approximation that allows us to scale our method to high-capacity neural network models.
  arXiv  Detail & Related papers  (2021-02-16T06:04:27Z)
• Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation [99.92568326314667]
  We propose the amortized conditional normalized maximum likelihood (ACNML) method as a scalable general-purpose approach for uncertainty estimation. Our algorithm builds on the conditional normalized maximum likelihood (CNML) coding scheme, which has minimax optimal properties according to the minimum description length principle. We demonstrate that ACNML compares favorably to a number of prior techniques for uncertainty estimation in terms of calibration on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-11-05T08:04:34Z)
• Balanced Meta-Softmax for Long-Tailed Visual Recognition [46.215759445665434]
  We show that the Softmax function, though used in most classification tasks, gives a biased gradient estimation under the long-tailed setup. This paper presents Balanced Softmax, an elegant unbiased extension of Softmax, to accommodate the label distribution shift between training and testing. In our experiments, we demonstrate that Balanced Meta-Softmax outperforms state-of-the-art long-tailed classification solutions on both visual recognition and instance segmentation tasks.
  arXiv  Detail & Related papers  (2020-07-21T12:05:00Z)
• Least Squares Regression with Markovian Data: Fundamental Limits and Algorithms [69.45237691598774]
  We study the problem of least squares linear regression where the data-points are dependent and are sampled from a Markov chain. We establish sharp information theoretic minimax lower bounds for this problem in terms of $tau_mathsfmix$. We propose an algorithm based on experience replay--a popular reinforcement learning technique--that achieves a significantly better error rate.
  arXiv  Detail & Related papers  (2020-06-16T04:26:50Z)
• Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations [154.51144248210338]
  Batch Nuclear-norm Maximization (BNM) is proposed to boost the learning under label insufficient learning scenarios. BNM outperforms competitors and works well with existing well-known methods.
  arXiv  Detail & Related papers  (2020-03-27T05:04:24Z)
• Being Bayesian about Categorical Probability [6.875312133832079]
  We consider a random variable of a categorical probability over class labels. In this framework, the prior distribution explicitly models the presumed noise inherent in the observed label. Our method can be implemented as a plug-and-play loss function with negligible computational overhead.
  arXiv  Detail & Related papers  (2020-02-19T02:35:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.