Orthogonal Uncertainty Representation of Data Manifold for Robust Long-Tailed Learning

• URL: http://arxiv.org/abs/2310.10090v1
• Date: Mon, 16 Oct 2023 05:50:34 GMT
• Title: Orthogonal Uncertainty Representation of Data Manifold for Robust Long-Tailed Learning
• Authors: Yanbiao Ma, Licheng Jiao, Fang Liu, Shuyuan Yang, Xu Liu, Lingling Li
• Abstract summary: In scenarios with long-tailed distributions, the model's ability to identify tail classes is limited due to the under-representation of tail samples. We propose an Orthogonal Uncertainty Representation (OUR) of feature embedding and an end-to-end training strategy to improve the long-tail phenomenon of model robustness.
• Score: 52.021899899683675
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In scenarios with long-tailed distributions, the model's ability to identify tail classes is limited due to the under-representation of tail samples. Class rebalancing, information augmentation, and other techniques have been proposed to facilitate models to learn the potential distribution of tail classes. The disadvantage is that these methods generally pursue models with balanced class accuracy on the data manifold, while ignoring the ability of the model to resist interference. By constructing noisy data manifold, we found that the robustness of models trained on unbalanced data has a long-tail phenomenon. That is, even if the class accuracy is balanced on the data domain, it still has bias on the noisy data manifold. However, existing methods cannot effectively mitigate the above phenomenon, which makes the model vulnerable in long-tailed scenarios. In this work, we propose an Orthogonal Uncertainty Representation (OUR) of feature embedding and an end-to-end training strategy to improve the long-tail phenomenon of model robustness. As a general enhancement tool, OUR has excellent compatibility with other methods and does not require additional data generation, ensuring fast and efficient training. Comprehensive evaluations on long-tailed datasets show that our method significantly improves the long-tail phenomenon of robustness, bringing consistent performance gains to other long-tailed learning methods.

Related papers

• Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss [0.4249842620609682]
  We build on the textitinvariant statistical loss (ISL) method introduced in citede2024training. We extend it to handle heavy-tailed and multivariate data distributions. We assess its performance in generative generative modeling and explore its potential as a pretraining technique for generative adversarial networks (GANs)
  arXiv  Detail & Related papers  (2024-10-29T10:27:50Z)
• FedLF: Adaptive Logit Adjustment and Feature Optimization in Federated Long-Tailed Learning [5.23984567704876]
  Federated learning offers a paradigm to the challenge of preserving privacy in distributed machine learning. Traditional approach fails to address the phenomenon of class-wise bias in global long-tailed data. New method FedLF introduces three modifications in the local training phase: adaptive logit adjustment, continuous class centred optimization, and feature decorrelation.
  arXiv  Detail & Related papers  (2024-09-18T16:25:29Z)
• Ask Your Distribution Shift if Pre-Training is Right for You [74.18516460467019]
  In practice, fine-tuning a pre-trained model improves robustness significantly in some cases but not at all in others. We focus on two possible failure modes of models under distribution shift: poor extrapolation and biases in the training data. Our study suggests that, as a rule of thumb, pre-training can help mitigate poor extrapolation but not dataset biases.
  arXiv  Detail & Related papers  (2024-02-29T23:46:28Z)
• FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning [21.693779973263172]
  In this paper, we introduce a fine-tuning approach termed Feature Discrimination Alignment (FD-Align) Our method aims to bolster the model's generalizability by preserving the consistency of spurious features. Once fine-tuned, the model can seamlessly integrate with existing methods, leading to performance improvements.
  arXiv  Detail & Related papers  (2023-10-23T17:12:01Z)
• Alleviating the Effect of Data Imbalance on Adversarial Training [26.36714114672729]
  We study adversarial training on datasets that obey the long-tailed distribution. We propose a new adversarial training framework -- Re-balancing Adversarial Training (REAT)
  arXiv  Detail & Related papers  (2023-07-14T07:01:48Z)
• Learning to Jump: Thinning and Thickening Latent Counts for Generative Modeling [69.60713300418467]
  Learning to jump is a general recipe for generative modeling of various types of data. We demonstrate when learning to jump is expected to perform comparably to learning to denoise, and when it is expected to perform better.
  arXiv  Detail & Related papers  (2023-05-28T05:38:28Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• Constructing Balance from Imbalance for Long-tailed Image Recognition [50.6210415377178]
  The imbalance between majority (head) classes and minority (tail) classes severely skews the data-driven deep neural networks. Previous methods tackle with data imbalance from the viewpoints of data distribution, feature space, and model design. We propose a concise paradigm by progressively adjusting label space and dividing the head classes and tail classes. Our proposed model also provides a feature evaluation method and paves the way for long-tailed feature learning.
  arXiv  Detail & Related papers  (2022-08-04T10:22:24Z)
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning [25.85044477227461]
  Models that are more accurate on the out-of-distribution data relative to this baseline exhibit "effective robustness" We find that models pre-trained on larger datasets exhibit effective robustness during training that vanishes at convergence. We discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.
  arXiv  Detail & Related papers  (2021-06-30T06:21:42Z)
• Long-tailed Recognition by Routing Diverse Distribution-Aware Experts [64.71102030006422]
  We propose a new long-tailed classifier called RoutIng Diverse Experts (RIDE) It reduces the model variance with multiple experts, reduces the model bias with a distribution-aware diversity loss, reduces the computational cost with a dynamic expert routing module. RIDE outperforms the state-of-the-art by 5% to 7% on CIFAR100-LT, ImageNet-LT and iNaturalist 2018 benchmarks.
  arXiv  Detail & Related papers  (2020-10-05T06:53:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.