Uncertainty-Aware Gradient Signal-to-Noise Data Selection for Instruction Tuning

• URL: http://arxiv.org/abs/2601.13697v1
• Date: Tue, 20 Jan 2026 07:51:32 GMT
• Title: Uncertainty-Aware Gradient Signal-to-Noise Data Selection for Instruction Tuning
• Abstract summary: Modern instruction datasets are large, noisy, and redundant, making full-data fine-tuning costly and unnecessary.<n>We propose GRADING, an objective-agnostic, uncertainty-aware data selection framework.<n>Our method matches or surpasses random subsets and strong baselines in most-as-a-judge evaluations and in human assessment.
• Score: 16.648609404000428
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Instruction tuning is a standard paradigm for adapting large language models (LLMs), but modern instruction datasets are large, noisy, and redundant, making full-data fine-tuning costly and often unnecessary. Existing data selection methods either build expensive gradient datastores or assign static scores from a weak proxy, largely ignoring evolving uncertainty, and thus missing a key source of LLM interpretability. We propose GRADFILTERING, an objective-agnostic, uncertainty-aware data selection framework that utilizes a small GPT-2 proxy with a LoRA ensemble and aggregates per-example gradients into a Gradient Signal-to-Noise Ratio (G-SNR) utility. Our method matches or surpasses random subsets and strong baselines in most LLM-as-a-judge evaluations as well as in human assessment. Moreover, GRADFILTERING-selected subsets converge faster than competitive filters under the same compute budget, reflecting the benefit of uncertainty-aware scoring.

Related papers

• Filter Learning for Subgraphs: Algebras and Performance Risk Bounds [63.46320825598287]
  We propose a framework for subgraph filter learning, where subgraph-supported operators approximate ambient graph filters under partial observations.<n>We develop a subgraph filter algebra based on distance-aware Laplacian constructions, defining a structured and controllable class of filters for effective approximation.
  arXiv  Detail & Related papers  (2026-07-23T12:36:05Z)
• Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels [4.826561433595647]
  We propose a bilevel optimization framework and prove, under certain assumptions, that it can recover the DPO optimum under clean data.<n>Considering that high-quality metadata may be difficult to obtain, we propose a task-agnostic meta-knowledge-driven method.<n> Experiments on TL;DR summarization and Anthropic HH single-turn dialogue show that the proposed method improves training performance over multiple DPO baselines under different noise rates.
  arXiv  Detail & Related papers  (2026-07-09T09:20:25Z)
• Unbiased Alignment for Large Language Models with Noisy Preferences [74.69408994904965]
  We present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss.<n>By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets.
  arXiv  Detail & Related papers  (2026-07-03T12:04:43Z)
• Self-Improving In-Context Learning [3.9202238580555417]
  We propose to improve in-context learning by optimizing the embeddings of a few prompt test at time.<n>We formalize this signal as a bounded, self-supervised calibration procedure.<n>Across a comprehensive suite of ICL tasks, the proposed calibration improves or matches upon the base model and consistently outperforms classification-specific baselines on most tasks.
  arXiv  Detail & Related papers  (2026-05-22T03:01:34Z)
• Beyond Static Best-of-N: Bayesian List-wise Alignment for LLM-based Recommendation [23.95623437107717]
  Large Language Models have revolutionized recommender systems (LLM4Rec) by leveraging their generative capabilities to model user preferences.<n>Existing LLM4Rec methods rely on token-level objectives, making it difficult to optimize list-level and non-differentiable metrics that define actual recommendation quality.<n>We propose BLADE (Bayesian List-wise Alignment via Dynamic Estimation) to overcome these challenges.
  arXiv  Detail & Related papers  (2026-05-06T07:02:57Z)
• Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling [40.94400211806987]
  We propose a policy-driven ZO framework that treats the sampling distribution over perturbation directions as a learnable policy.<n>We show that learned sampling improves quality gradient information and relax the explicit dependence on $d$ in convergence bounds.<n>Our results suggest that adaptive direction sampling is a promising route to make ZO fine-tuning viable at scale.
  arXiv  Detail & Related papers  (2026-02-14T08:01:41Z)
• Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models [52.48582333951919]
  We propose a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates.<n>SAGE (Stability-Aware Gradient Efficiency) integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines.
  arXiv  Detail & Related papers  (2026-02-01T12:56:10Z)
• LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection [32.35731324386828]
  Adapting large language models (LLMs) to specific domains often faces a critical bottleneck: the scarcity of high-quality, human-curated data.<n>Existing approaches, categorized as similarity-based and direct optimization methods, struggle to simultaneously achieve these goals.<n>We introduce LAMDAS, a novel approach that leverages the pre-trained LLM itself as an implicit classifier.
  arXiv  Detail & Related papers  (2025-09-08T10:30:58Z)
• LLM Unlearning using Gradient Ratio-Based Influence Estimation and Noise Injection [0.0]
  Existing empirical methods often yield incomplete forgetting or unintended degradation of unrelated knowledge due to poor localization.<n>GRIN introduces a novel gradient-ratio-based metric to identify parameters most responsible for memorizing forget data.<n>We then perform selective noise injection into these parameters prior to fine-tuning, which improves unlearning performance while maintaining model utility.
  arXiv  Detail & Related papers  (2025-08-08T17:15:32Z)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs [63.32585910975191]
  We introduce reward-conditioned Large Language Models (LLMs) that learn from the entire spectrum of response quality within the dataset.<n>We show that our approach consistently boosts DPO by a considerable margin.<n>Our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere data expansion.
  arXiv  Detail & Related papers  (2024-10-10T16:01:51Z)
• Stable Neighbor Denoising for Source-free Domain Adaptive Segmentation [91.83820250747935]
  Pseudo-label noise is mainly contained in unstable samples in which predictions of most pixels undergo significant variations during self-training. We introduce the Stable Neighbor Denoising (SND) approach, which effectively discovers highly correlated stable and unstable samples. SND consistently outperforms state-of-the-art methods in various SFUDA semantic segmentation settings.
  arXiv  Detail & Related papers  (2024-06-10T21:44:52Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Automated classification of pre-defined movement patterns: A comparison between GNSS and UWB technology [55.41644538483948]
  Real-time location systems (RTLS) allow for collecting data from human movement patterns. The current study aims to design and evaluate an automated framework to classify human movement patterns in small areas.
  arXiv  Detail & Related papers  (2023-03-10T14:46:42Z)
• MAPS: A Noise-Robust Progressive Learning Approach for Source-Free Domain Adaptive Keypoint Detection [76.97324120775475]
  Cross-domain keypoint detection methods always require accessing the source data during adaptation. This paper considers source-free domain adaptive keypoint detection, where only the well-trained source model is provided to the target domain.
  arXiv  Detail & Related papers  (2023-02-09T12:06:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.