Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks

• URL: http://arxiv.org/abs/2604.12945v1
• Date: Tue, 14 Apr 2026 16:41:33 GMT
• Title: Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks
• Abstract summary: Deep neural networks are typically trained by uniformly sampling large datasets across epochs.<n>Recent work shows that progressively reducing the amount of training data can improve efficiency and generalization.<n>We propose Adaptive Data Dropout, a framework that dynamically adjusts the subset of training data based on performance feedback.
• Score: 10.269961654607108
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks are typically trained by uniformly sampling large datasets across epochs, despite evidence that not all samples contribute equally throughout learning. Recent work shows that progressively reducing the amount of training data can improve efficiency and generalization, but existing methods rely on fixed schedules that do not adapt during training. In this work, we propose Adaptive Data Dropout, a simple framework that dynamically adjusts the subset of training data based on performance feedback. Inspired by self-regulated learning, our approach treats data selection as an adaptive process, increasing or decreasing data exposure in response to changes in training accuracy. We introduce a lightweight stochastic update mechanism that modulates the dropout schedule online, allowing the model to balance exploration and consolidation over time. Experiments on standard image classification benchmarks show that our method reduces effective training steps while maintaining competitive accuracy compared to static data dropout strategies. These results highlight adaptive data selection as a promising direction for efficient and robust training. Code will be released.

Related papers

• Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods [30.96121293915128]
  ADAPT is a dynamic online framework that reweights training samples with adaptive per-sample learning rates guided by similarity-based quality signals.<n> ADAPT consistently outperforms offline selection/mixing and prior online methods, achieving stronger cross-benchmark generalization under equal FLOPs.
  arXiv  Detail & Related papers  (2026-04-19T14:23:23Z)
• MetaDAT: Generalizable Trajectory Prediction via Meta Pre-training and Data-Adaptive Test-Time Updating [18.84791793147564]
  Existing trajectory prediction methods exhibit significant performance degradation under distribution shifts during test time.<n>We propose a meta-learning framework to directly optimize the predictor for fast and accurate online adaptation.<n>At test time, we introduce a data-adaptive model updating mechanism that dynamically adjusts the learning rates and updating frequencies based on online partial derivatives and hard sample selection.<n>Results demonstrate that our method achieves superior adaptation accuracy, surpassing state-of-the-art test-time training methods for trajectory prediction.
  arXiv  Detail & Related papers  (2026-03-10T09:34:32Z)
• Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data [89.96277093034547]
  We introduce EntroDrop, an entropy-guided token dropout method that functions as structured data regularization.<n>We show that EntroDrop consistently outperforms standard regularization baselines and maintains robust performance throughout extended multi-epoch training.
  arXiv  Detail & Related papers  (2025-12-29T12:35:51Z)
• Capturing the Temporal Dependence of Training Data Influence [100.91355498124527]
  We formalize the concept of trajectory-specific leave-one-out influence, which quantifies the impact of removing a data point during training.<n>We propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO.<n>As data value embedding captures training data ordering, it offers valuable insights into model training dynamics.
  arXiv  Detail & Related papers  (2024-12-12T18:28:55Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Distilled Datamodel with Reverse Gradient Matching [74.75248610868685]
  We introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages. Our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.
  arXiv  Detail & Related papers  (2024-04-22T09:16:14Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Efficient Augmentation for Imbalanced Deep Learning [8.38844520504124]
  We study a convolutional neural network's internal representation of imbalanced image data. We measure the generalization gap between a model's feature embeddings in the training and test sets, showing that the gap is wider for minority classes. This insight enables us to design an efficient three-phase CNN training framework for imbalanced data.
  arXiv  Detail & Related papers  (2022-07-13T09:43:17Z)
• CAFA: Class-Aware Feature Alignment for Test-Time Adaptation [50.26963784271912]
  Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time. We propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously encourages a model to learn target representations in a class-discriminative manner.
  arXiv  Detail & Related papers  (2022-06-01T03:02:07Z)
• Friendly Training: Neural Networks Can Adapt Data To Make Learning Easier [23.886422706697882]
  We propose a novel training procedure named Friendly Training. We show that Friendly Training yields improvements with respect to informed data sub-selection and random selection. Results suggest that adapting the input data is a feasible way to stabilize learning and improve the skills generalization of the network.
  arXiv  Detail & Related papers  (2021-06-21T10:50:34Z)
• Online Coreset Selection for Rehearsal-based Continual Learning [65.85595842458882]
  In continual learning, we store a subset of training examples (coreset) to be replayed later to alleviate catastrophic forgetting. We propose Online Coreset Selection (OCS), a simple yet effective method that selects the most representative and informative coreset at each iteration. Our proposed method maximizes the model's adaptation to a target dataset while selecting high-affinity samples to past tasks, which directly inhibits catastrophic forgetting.
  arXiv  Detail & Related papers  (2021-06-02T11:39:25Z)
• Gradient-guided Loss Masking for Neural Machine Translation [27.609155878513334]
  In this paper, we explore strategies that dynamically optimize data usage during the training process. Our algorithm calculates the gradient alignment between the training data and the clean data to mask out data with negative alignment. Experiments on three WMT language pairs show that our method brings significant improvement over strong baselines.
  arXiv  Detail & Related papers  (2021-02-26T15:41:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.