How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients

• URL: http://arxiv.org/abs/2504.10766v1
• Date: Mon, 14 Apr 2025 23:53:47 GMT
• Title: How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients
• Authors: Ming Li, Yanhong Li, Ziyue Li, Tianyi Zhou,
• Abstract summary: We present a spectral analysis of layer-wise gradients induced by low/high-quality instruction and reasoning data.<n>Our work illuminates the interplay between data quality and training stability, shedding novel insights into developing better data exploration strategies for post-training.
• Score: 24.699470675718466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As the post-training of large language models (LLMs) advances from instruction-following to complex reasoning tasks, understanding how different data affect finetuning dynamics remains largely unexplored. In this paper, we present a spectral analysis of layer-wise gradients induced by low/high-quality instruction and reasoning data for LLM post-training. Our analysis reveals that widely-studied metrics for data evaluation, e.g., IFD, InsTag, Difficulty, and Reward, can be explained and unified by spectral properties computed from gradients' singular value decomposition (SVD). Specifically, higher-quality data are usually associated with lower nuclear norms and higher effective ranks. Notably, effective rank exhibits better robustness and resolution than nuclear norm in capturing subtle quality differences. For example, reasoning data achieves substantially higher effective ranks than instruction data, implying richer gradient structures on more complex tasks. Our experiments also highlight that models within the same family share similar gradient patterns regardless of their sizes, whereas different model families diverge significantly. Providing a unified view on the effects of data quality across instruction and reasoning data, this work illuminates the interplay between data quality and training stability, shedding novel insights into developing better data exploration strategies for post-training.

Related papers

• Clear Minds Think Alike: What Makes LLM Fine-tuning Robust? A Study of Token Perplexity [61.48338027901318]
  We show that fine-tuning with LLM-generated data improves target task performance and reduces out-of-domain degradation. This is the first mechanistic explanation for the superior OOD robustness conferred by LLM-generated training data.
  arXiv  Detail & Related papers  (2025-01-24T08:18:56Z)
• Towards Robust Out-of-Distribution Generalization: Data Augmentation and Neural Architecture Search Approaches [4.577842191730992]
  We study ways toward robust OoD generalization for deep learning. We first propose a novel and effective approach to disentangle the spurious correlation between features that are not essential for recognition. We then study the problem of strengthening neural architecture search in OoD scenarios.
  arXiv  Detail & Related papers  (2024-10-25T20:50:32Z)
• Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning [37.54523122932728]
  We propose a pipeline-based data augmentation method via large language models (LLMs) To tackle the issue of low data diversity, our pipeline utilizes knowledge graphs (KGs) to extract entities and quantities. To address high data noise, the GCSE model uses a Gaussian-decayed function to limit the impact of false hard negative samples.
  arXiv  Detail & Related papers  (2024-09-19T16:29:58Z)
• G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translation [21.506844286376275]
  We propose a novel gradient-based method to automatically select high-quality and diverse instruction finetuning data for machine translation. Our key innovation centers around analyzing how individual training examples influence the model during training.
  arXiv  Detail & Related papers  (2024-05-21T16:38:13Z)
• LESS: Selecting Influential Data for Targeted Instruction Tuning [64.78894228923619]
  We propose LESS, an efficient algorithm to estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. We show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Our method goes beyond surface form cues to identify data that the necessary reasoning skills for the intended downstream application.
  arXiv  Detail & Related papers  (2024-02-06T19:18:04Z)
• Hodge-Aware Contrastive Learning [101.56637264703058]
  Simplicial complexes prove effective in modeling data with multiway dependencies. We develop a contrastive self-supervised learning approach for processing simplicial data.
  arXiv  Detail & Related papers  (2023-09-14T00:40:07Z)
• Robust Learning with Progressive Data Expansion Against Spurious Correlation [65.83104529677234]
  We study the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. We propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance.
  arXiv  Detail & Related papers  (2023-06-08T05:44:06Z)
• Diving into Unified Data-Model Sparsity for Class-Imbalanced Graph Representation Learning [30.23894624193583]
  Graph Neural Networks (GNNs) training upon non-Euclidean graph data often encounters relatively higher time costs. We develop a unified data-model dynamic sparsity framework named Graph Decantation (GraphDec) to address challenges brought by training upon a massive class-imbalanced graph data.
  arXiv  Detail & Related papers  (2022-10-01T01:47:00Z)
• The Manifold Hypothesis for Gradient-Based Explanations [55.01671263121624]
  gradient-based explanation algorithms provide perceptually-aligned explanations. We show that the more a feature attribution is aligned with the tangent space of the data, the more perceptually-aligned it tends to be. We suggest that explanation algorithms should actively strive to align their explanations with the data manifold.
  arXiv  Detail & Related papers  (2022-06-15T08:49:24Z)
• CHALLENGER: Training with Attribution Maps [63.736435657236505]
  We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance. In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
  arXiv  Detail & Related papers  (2022-05-30T13:34:46Z)
• Self-learn to Explain Siamese Networks Robustly [22.913886901196353]
  Learning to compare two objects are used in digital forensics, face recognition, brain network analysis, especially when labeled data is scarce. As these applications make high-stake decisions involving societal values like fairness and imbalance, it is critical to explain learned models.
  arXiv  Detail & Related papers  (2021-09-15T15:28:39Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.