On the Difficulty of Learning a Meta-network for Training Data Selection

• URL: http://arxiv.org/abs/2606.00571v1
• Date: Sat, 30 May 2026 06:47:24 GMT
• Title: On the Difficulty of Learning a Meta-network for Training Data Selection
• Abstract summary: We show that Meta-learning for Training-data Selection (MTS) often performs below expectation.<n>We present a mathematical analysis of MTS, which reveals the dynamics of normalized data weights and the relation between disparate data quality and poor GSNR.<n>We propose a set of informative features that capture the positions of training data in their distributions and training dynamics.
• Score: 1.9431149786214235
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthetic data are increasingly used to train neural networks, yet distributional mismatch with real data limits their effectiveness when used indiscriminately. A common strategy is to learn data weights via bi-level optimization, which we refer to as Meta-learning for Training-data Selection (MTS). Interestingly, in practice, MTS often performs below expectation. We identify two obstacles in properly training MTS: a poor gradient signal-to-noise ratio (GSNR), which causes optimization difficulties, and lack of informative features that correlates with data quality. We present a mathematical analysis of MTS, which reveals the dynamics of normalized data weights and the relation between disparate data quality and poor GSNR. The analysis suggests a a simple yet effective solution: increasing the batch size. Further, we propose a set of informative features that capture the positions of training data in their distributions and training dynamics. Experiments across four benchmarks show consistent improvements, achieving average gains of 5.49% over training without selection and 2.89% over the strongest baseline.

Related papers

• Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning [7.431998098521794]
  We introduce the Holistic Data Scheduler (HDS), a novel online data mixing framework.<n>HDS formulates the data scheduling challenge as a reinforcement learning problem in a continuous control space.<n>It integrates a data-driven reward for quality, a loss-driven reward capturing inter-domain influence, and a model-driven reward based on weight norms.
  arXiv  Detail & Related papers  (2026-06-23T04:32:46Z)
• Data Agent: Learning to Select Data via End-to-End Dynamic Optimization [37.1771265765151]
  Data Agent formulates data selection as a training-aware sequential decision-making problem.<n>Experiments demonstrate that Data Agent consistently accelerates training while preserving or improving performance.<n>Its dataset-agnostic formulation and modular reward make it plug-and-play across tasks and scenarios.
  arXiv  Detail & Related papers  (2026-03-08T03:10:39Z)
• Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward [54.708851958671794]
  We propose a Data-Efficient Policy Optimization pipeline that combines optimized strategies for both offline and online data selection.<n>In offline phase, we curate a high-quality subset of training samples based on diversity, influence, and appropriate difficulty.<n>During online RLVR training, we introduce a sample-level explorability metric to dynamically filter samples with low exploration potential.
  arXiv  Detail & Related papers  (2025-09-01T10:04:20Z)
• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining [55.262510814326035]
  Existing reweighting strategies primarily focus on group-level data importance.<n>We introduce novel algorithms for dynamic, instance-level data reweighting.<n>Our framework allows us to devise reweighting strategies deprioritizing redundant or uninformative data.
  arXiv  Detail & Related papers  (2025-02-10T17:57:15Z)
• MetaTrading: An Immersion-Aware Model Trading Framework for Vehicular Metaverse Services [92.40586697273868]
  Timely updating of Internet of Things data is crucial for achieving immersion in vehicular metaverse services.<n>We propose an immersion-aware model trading framework that enables efficient and privacy-preserving data provisioning through federated learning.<n> Experimental results show that the proposed framework outperforms state-of-the-art benchmarks.
  arXiv  Detail & Related papers  (2024-10-25T16:20:46Z)
• Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging [11.223074654129915]
  Supervised fine-tuning (SFT) is crucial for adapting Large Language Models (LLMs) to specific tasks. We propose to mitigate this imbalance by merging SFT models fine-tuned with different data orders.
  arXiv  Detail & Related papers  (2024-10-01T08:44:31Z)
• Efficient Online Data Mixing For Language Model Pre-Training [101.45242332613944]
  Existing data selection methods suffer from slow and computationally expensive processes. Data mixing, on the other hand, reduces the complexity of data selection by grouping data points together. We develop an efficient algorithm for Online Data Mixing (ODM) that combines elements from both data selection and data mixing.
  arXiv  Detail & Related papers  (2023-12-05T00:42:35Z)
• A Study on the Impact of Data Augmentation for Training Convolutional Neural Networks in the Presence of Noisy Labels [14.998309259808236]
  Label noise is common in large real-world datasets, and its presence harms the training process of deep neural networks. We evaluate the impact of data augmentation as a design choice for training deep neural networks. We show that the appropriate selection of data augmentation can drastically improve the model robustness to label noise.
  arXiv  Detail & Related papers  (2022-08-23T20:04:17Z)
• CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning [55.733193075728096]
  Modern deep neural networks can easily overfit to biased training data containing corrupted labels or class imbalance. Sample re-weighting methods are popularly used to alleviate this data bias issue. We propose a meta-model capable of adaptively learning an explicit weighting scheme directly from data.
  arXiv  Detail & Related papers  (2022-02-11T13:49:51Z)
• CatFedAvg: Optimising Communication-efficiency and Classification Accuracy in Federated Learning [2.2172881631608456]
  We introduce a new family of Federated Learning algorithms called CatFedAvg. It improves the communication efficiency but improves the quality of learning using a category coverage inNIST strategy. Our experiments show that an increase of 10% absolute points accuracy using the M dataset with 70% absolute points lower network transfer over FedAvg.
  arXiv  Detail & Related papers  (2020-11-14T06:52:02Z)
• Meta Feature Modulator for Long-tailed Recognition [37.90990378643794]
  We propose a meta-learning framework to model the difference between the long-tailed training data and the balanced meta data from the perspective of representation learning. We further design a modulator network to guide the generation of the modulation parameters, and such a meta-learner can be readily adapted to train the classification network on other long-tailed datasets.
  arXiv  Detail & Related papers  (2020-08-08T03:19:03Z)
• COLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating Minimization [60.07531696857743]
  Co-Learns DNNs and soft labels through Alternating Minimization of two objectives. We propose COLAM framework that Co-Learns DNNs and soft labels through Alternating Minimization of two objectives.
  arXiv  Detail & Related papers  (2020-04-26T17:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.