Adaptively Point-weighting Curriculum Learning

• URL: http://arxiv.org/abs/2505.01665v1
• Date: Sat, 03 May 2025 03:27:22 GMT
• Title: Adaptively Point-weighting Curriculum Learning
• Authors: Wensheng Li, Hao Wang, Ruifeng Zhou, Hanting Guan, Chao Zhang, Dacheng Tao,
• Abstract summary: Curriculum learning (CL) is a training strategy that makes easy samples learned first and then fits hard samples.<n>We develop the adaptively point-weighting (APW) curriculum learning algorithm, which adaptively assigns the weight to every training sample.<n>We present the theoretical analysis on the properties of APW including training effectiveness, training feasibility, training stability, and generalization performance.
• Score: 46.757954778447505
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Curriculum learning (CL) is referred to as a training strategy that makes easy samples learned first and then fits hard samples. It imitates the process of humans learning knowledge, and has become a potential manner of effectively training deep networks. In this study, we develop the adaptively point-weighting (APW) curriculum learning algorithm, which adaptively assigns the weight to every training sample not only based on its training error but also considering the current training state of the network. Specifically, in the early training phase, it increases the weights of easy samples to make the network rapidly capture the overall characteristics of the dataset; and in the later training phase, the weights of hard points rise to improve the fitting performance on the discrete local regions. Moreover, we also present the theoretical analysis on the properties of APW including training effectiveness, training feasibility, training stability, and generalization performance. The numerical experiments support the superiority of APW and demonstrate the validity of our theoretical findings.

Related papers

• EvoLM: In Search of Lost Language Model Training Dynamics [97.69616550374579]
  EvoLM is a model suite that enables systematic and transparent analysis of LMs' training dynamics across pre-training, continued pre-training, supervised fine-tuning, and reinforcement learning.<n>By training over 100 LMs with 1B and 4B parameters from scratch, we rigorously evaluate both upstream (language modeling) and downstream (problem-solving) reasoning capabilities.
  arXiv  Detail & Related papers  (2025-06-19T04:58:47Z)
• CLASS: Enhancing Cross-Modal Text-Molecule Retrieval Performance and Training Efficiency [7.2360149365370345]
  Cross-modal text-molecule retrieval task bridges molecule structures and natural language descriptions.<n>Existing methods predominantly focus on aligning text modality and molecule modality, yet they overlook adaptively adjusting the learning states at different training stages.<n>This paper proposes a Curriculum Learning-bAsed croSS-modal text-molecule training framework (CLASS), which can be integrated with any backbone to yield promising performance improvement.
  arXiv  Detail & Related papers  (2025-02-17T10:24:07Z)
• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining [55.262510814326035]
  Existing reweighting strategies primarily focus on group-level data importance.<n>We introduce novel algorithms for dynamic, instance-level data reweighting.<n>Our framework allows us to devise reweighting strategies deprioritizing redundant or uninformative data.
  arXiv  Detail & Related papers  (2025-02-10T17:57:15Z)
• BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping [64.8477128397529]
  We propose a training-required and training-free test-time adaptation framework. We maintain a light-weight key-value memory for feature retrieval from instance-agnostic historical samples and instance-aware boosting samples. We theoretically justify the rationality behind our method and empirically verify its effectiveness on both the out-of-distribution and the cross-domain datasets.
  arXiv  Detail & Related papers  (2024-10-20T15:58:43Z)
• Effective Vision Transformer Training: A Data-Centric Perspective [24.02488085447691]
  Vision Transformers (ViTs) have shown promising performance compared with Convolutional Neural Networks (CNNs) In this paper, we define several metrics, including Dynamic Data Proportion (DDP) and Knowledge Assimilation Rate (KAR) We propose a novel data-centric ViT training framework to dynamically measure the difficulty'' of training samples and generate effective'' samples for models at different training stages.
  arXiv  Detail & Related papers  (2022-09-29T17:59:46Z)
• Towards Balanced Learning for Instance Recognition [149.76724446376977]
  We propose Libra R-CNN, a framework towards balanced learning for instance recognition. It integrates IoU-balanced sampling, balanced feature pyramid, and objective re-weighting, respectively for reducing the imbalance at sample, feature, and objective level.
  arXiv  Detail & Related papers  (2021-08-23T13:40:45Z)
• Friendly Training: Neural Networks Can Adapt Data To Make Learning Easier [23.886422706697882]
  We propose a novel training procedure named Friendly Training. We show that Friendly Training yields improvements with respect to informed data sub-selection and random selection. Results suggest that adapting the input data is a feasible way to stabilize learning and improve the skills generalization of the network.
  arXiv  Detail & Related papers  (2021-06-21T10:50:34Z)
• Self-Adaptive Training: Bridging the Supervised and Self-Supervised Learning [16.765461276790944]
  Self-adaptive training is a unified training algorithm that dynamically calibrates and enhances training process by model predictions without incurring extra computational cost. We analyze the training dynamics of deep networks on training data corrupted by, e.g., random noise and adversarial examples. Our analysis shows that model predictions are able to magnify useful underlying information in data and this phenomenon occurs broadly even in the absence of emphany label information.
  arXiv  Detail & Related papers  (2021-01-21T17:17:30Z)
• Efficient Estimation of Influence of a Training Instance [56.29080605123304]
  We propose an efficient method for estimating the influence of a training instance on a neural network model. Our method is inspired by dropout, which zero-masks a sub-network and prevents the sub-network from learning each training instance. We demonstrate that the proposed method can capture training influences, enhance the interpretability of error predictions, and cleanse the training dataset for improving generalization.
  arXiv  Detail & Related papers  (2020-12-08T04:31:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.