Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification

• URL: http://arxiv.org/abs/2410.03588v1
• Date: Fri, 4 Oct 2024 16:47:11 GMT
• Title: Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification
• Authors: Kelsey Lieberman, Swarna Kamlam Ravindran, Shuai Yuan, Carlo Tomasi,
• Abstract summary: Conditional Loss Training (LCT) can be used to train reliable classifiers under severe class imbalance. We show that LCT approximates the performance of several models and improves the overall performance of models on both CIFAR and real medical imaging applications.
• Score: 3.06506506650274
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although binary classification is a well-studied problem, training reliable classifiers under severe class imbalance remains a challenge. Recent techniques mitigate the ill effects of imbalance on training by modifying the loss functions or optimization methods. We observe that different hyperparameter values on these loss functions perform better at different recall values. We propose to exploit this fact by training one model over a distribution of hyperparameter values--instead of a single value--via Loss Conditional Training (LCT). Experiments show that training over a distribution of hyperparameters not only approximates the performance of several models but actually improves the overall performance of models on both CIFAR and real medical imaging applications, such as melanoma and diabetic retinopathy detection. Furthermore, training models with LCT is more efficient because some hyperparameter tuning can be conducted after training to meet individual needs without needing to retrain from scratch.

Related papers

• SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation [52.6922833948127]
  In this work, we investigate the importance of parameters in pre-trained diffusion models. We propose a novel model fine-tuning method to make full use of these ineffective parameters. Our method enhances the generative capabilities of pre-trained models in downstream applications.
  arXiv  Detail & Related papers  (2024-09-10T16:44:47Z)
• Criticality Leveraged Adversarial Training (CLAT) for Boosted Performance via Parameter Efficiency [15.211462468655329]
  CLAT introduces parameter efficiency into the adversarial training process, improving both clean accuracy and adversarial robustness. It can be applied on top of existing adversarial training methods, significantly reducing the number of trainable parameters by approximately 95%.
  arXiv  Detail & Related papers  (2024-08-19T17:58:03Z)
• Probing Perfection: The Relentless Art of Meddling for Pulmonary Airway Segmentation from HRCT via a Human-AI Collaboration Based Active Learning Method [13.384578466263566]
  In pulmonary tracheal segmentation, the scarcity of annotated data is a prevalent issue. Deep Learning (DL) methods face challenges: the opacity of 'black box' models and the need for performance enhancement. We address these challenges by combining diverse query strategies with various DL models.
  arXiv  Detail & Related papers  (2024-07-03T23:27:53Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Optimizing for ROC Curves on Class-Imbalanced Data by Training over a Family of Loss Functions [3.06506506650274]
  Training reliable classifiers under severe class imbalance is a challenging problem in computer vision. Recent work has proposed techniques that mitigate the effects of training under imbalance by modifying the loss functions or optimization methods. We propose training over a family of loss functions, instead of a single loss function.
  arXiv  Detail & Related papers  (2024-02-08T04:31:21Z)
• Robust Learning with Progressive Data Expansion Against Spurious Correlation [65.83104529677234]
  We study the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. We propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance.
  arXiv  Detail & Related papers  (2023-06-08T05:44:06Z)
• One-stop Training of Multiple Capacity Models [74.87789190840527]
  We propose a novel one-stop training framework to jointly train high-capacity and low-capactiy models. Unlike knowledge distillation, where multiple capacity models are trained from scratch separately, our approach integrates supervisions from different capacity models simultaneously.
  arXiv  Detail & Related papers  (2023-05-23T13:44:09Z)
• Density-Aware Personalized Training for Risk Prediction in Imbalanced Medical Data [89.79617468457393]
  Training models with imbalance rate (class density discrepancy) may lead to suboptimal prediction. We propose a framework for training models for this imbalance issue. We demonstrate our model's improved performance in real-world medical datasets.
  arXiv  Detail & Related papers  (2022-07-23T00:39:53Z)
• A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustness [11.35810118757863]
  Existing models that achieve state-of-the-art (SOTA) performance on both clean and adversarially-perturbed images rely on convolution operations conditioned with feature-wise linear modulation (FiLM) layers. We present a fast learnable once-for-all adversarial training (FLOAT) algorithm, which instead of the existing FiLM-based conditioning, presents a unique weight conditioned learning that requires no additional layer. In particular, we add scaled noise to the weight tensors that enables a trade-off between clean and adversarial performance.
  arXiv  Detail & Related papers  (2022-03-28T19:25:36Z)
• FairIF: Boosting Fairness in Deep Learning via Influence Functions with Validation Set Sensitive Attributes [51.02407217197623]
  We propose a two-stage training algorithm named FAIRIF. It minimizes the loss over the reweighted data set where the sample weights are computed. We show that FAIRIF yields models with better fairness-utility trade-offs against various types of bias.
  arXiv  Detail & Related papers  (2022-01-15T05:14:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.