GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection

• URL: http://arxiv.org/abs/2603.09865v1
• Date: Tue, 10 Mar 2026 16:28:48 GMT
• Title: GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection
• Abstract summary: Gradient-aligned Sparse Tuning (GAST) is an innovative method that simultaneously performs selective fine-tuning at both data and layer dimensions.<n>GAST specifically targets redundancy in information by employing a layer-sparse strategy that adaptively selects the most impactful data points for each layer.
• Score: 12.622280605476256
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) has become a key strategy for adapting large language models, with recent advances in sparse tuning reducing overhead by selectively updating key parameters or subsets of data. Existing approaches generally focus on two distinct paradigms: layer-selective methods aiming to fine-tune critical layers to minimize computational load, and data-selective methods aiming to select effective training subsets to boost training. However, current methods typically overlook the fact that different data points contribute varying degrees to distinct model layers, and they often discard potentially valuable information from data perceived as of low quality. To address these limitations, we propose Gradient-aligned Sparse Tuning (GAST), an innovative method that simultaneously performs selective fine-tuning at both data and layer dimensions as integral components of a unified optimization strategy. GAST specifically targets redundancy in information by employing a layer-sparse strategy that adaptively selects the most impactful data points for each layer, providing a more comprehensive and sophisticated solution than approaches restricted to a single dimension. Experiments demonstrate that GAST consistently outperforms baseline methods, establishing a promising direction for future research in PEFT strategies.

Related papers

• SLAP: Stratified Loss-based Pruning for On-Policy Data-Efficient Instruction Tuning [14.87050258251449]
  We propose textbfSLAP, a novel batch-aware data selection framework.<n>SLAP evaluates the learnability of entire batch compositions rather than individual.<n>SLAP achieves superior performance with 20-40% less training data compared to full dataset training.
  arXiv  Detail & Related papers  (2026-05-13T06:36:24Z)
• Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxies [50.39041754816285]
  We propose a framework for learning multi-validation weights that jointly adapts data selection to both the downstream task and the specific model.<n>Our approach achieves performance comparable to or exceeding full-dataset tuning while using only 30% of the training samples on GSM8K.
  arXiv  Detail & Related papers  (2026-05-10T17:30:16Z)
• Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations [98.44542103979735]
  First-order gradient descent methods, epitomized by gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines.<n>Large-scale model training, stringent privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency.<n>We retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstreams across diverse model architectures and training scenarios.<n>We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research.
  arXiv  Detail & Related papers  (2026-04-14T17:01:36Z)
• MIBoost: A Gradient Boosting Algorithm for Variable Selection After Multiple Imputation [0.0]
  In practice, analyses are often complicated by missing data.<n>We propose MIBoost, a novel algorithm that employs a uniform variable-selection mechanism across imputed datasets.
  arXiv  Detail & Related papers  (2025-07-29T13:42:38Z)
• LASER: Stratified Selective Sampling for Instruction Tuning with Dedicated Scoring Strategy [5.130520244165049]
  We show that data selection can be both -- efficient and universal -- by using a multi-step pipeline.<n>We use task-based categorization to control the composition of our final data.<n>This integrated strategy enables high-performance fine-tuning with minimal overhead.
  arXiv  Detail & Related papers  (2025-05-28T09:22:25Z)
• Adaptive Dataset Quantization [2.0105434963031463]
  We introduce a versatile framework for dataset compression, namely Adaptive dataset Quantization (ADQ)<n>We propose a novel adaptive sampling strategy through the evaluation of generated bins' representativeness score, diversity score and importance score.<n>Our method not only exhibits superior generalization capability across different architectures, but also attains state-of-the-art results, surpassing DQ by average 3% on various datasets.
  arXiv  Detail & Related papers  (2024-12-22T07:08:29Z)
• Transfer Learning in $\ell_1$ Regularized Regression: Hyperparameter Selection Strategy based on Sharp Asymptotic Analysis [3.5374094795720854]
  Transfer learning techniques aim to leverage information from multiple related datasets to enhance prediction quality against a target dataset.<n>Some Lasso-based algorithms have been invented: Trans-Lasso and Pretraining Lasso.<n>We conduct a thorough, precise study of the algorithm in a high-dimensional setting via an analysis using the replica method.<n>Our approach reveals a surprisingly simple behavior of the algorithm: Ignoring one of the two types of information transferred to the fine-tuning stage has little effect on generalization performance.
  arXiv  Detail & Related papers  (2024-09-26T10:20:59Z)
• Exploring Selective Layer Fine-Tuning in Federated Learning [48.470385357429215]
  Federated learning (FL) has emerged as a promising paradigm for fine-tuning foundation models using distributed data. We study selective layer fine-tuning in FL, emphasizing a flexible approach that allows the clients to adjust their selected layers according to their local data and resources.
  arXiv  Detail & Related papers  (2024-08-28T07:48:39Z)
• LESS: Selecting Influential Data for Targeted Instruction Tuning [64.78894228923619]
  We propose LESS, an efficient algorithm to estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. We show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Our method goes beyond surface form cues to identify data that the necessary reasoning skills for the intended downstream application.
  arXiv  Detail & Related papers  (2024-02-06T19:18:04Z)
• Large-scale Fully-Unsupervised Re-Identification [78.47108158030213]
  We propose two strategies to learn from large-scale unlabeled data. The first strategy performs a local neighborhood sampling to reduce the dataset size in each without violating neighborhood relationships. A second strategy leverages a novel Re-Ranking technique, which has a lower time upper bound complexity and reduces the memory complexity from O(n2) to O(kn) with k n.
  arXiv  Detail & Related papers  (2023-07-26T16:19:19Z)
• Dynamic Scale Training for Object Detection [111.33112051962514]
  We propose a Dynamic Scale Training paradigm (abbreviated as DST) to mitigate scale variation challenge in object detection. Experimental results demonstrate the efficacy of our proposed DST towards scale variation handling. It does not introduce inference overhead and could serve as a free lunch for general detection configurations.
  arXiv  Detail & Related papers  (2020-04-26T16:48:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.