One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning

• URL: http://arxiv.org/abs/2605.06166v1
• Date: Thu, 07 May 2026 12:52:02 GMT
• Title: One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning
• Abstract summary: In Large Language Model (LLM) fine-tuning, parameter and data selection are common strategies for reducing fine-tuning cost.<n>We propose DualSFT (Dual-Selection Fine-Tuning), a one-shot dual-scoring algorithm that produces a parameter mask and data subset from shared gradient statistics.<n>On 3B-9B LLMs, single-axis DualSFT variants strengthen target-task performance and stability-plasticity trade-offs within comparison groups, while full DualSFT yields a more favorable joint-constrained trade-off.
• Score: 16.233534918614417
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In Large Language Model (LLM) fine-tuning, parameter and data selection are common strategies for reducing fine-tuning cost, yet they are typically driven by separate scoring mechanisms. When a parameter mask and data subset jointly determine restricted fine-tuning, this separation incurs redundant overhead and makes coordinated selection difficult. We cast parameter and data selection as two bilevel selection problems under a common validation objective and derive a shared local response-surrogate scoring rule. Under first- and second-order validation-improvement approximations, parameter importance and data utility emerge as column-wise and row-wise aggregations of a single gradient interaction matrix, yielding a closed-form row-column correspondence for co-extracting both signals. Building on this structure, we propose DualSFT (Dual-Selection Fine-Tuning), a one-shot dual-scoring algorithm that produces a parameter mask and data subset from shared gradient statistics. On 3B-9B LLMs, single-axis DualSFT variants strengthen target-task performance and stability-plasticity trade-offs within their comparison groups, while full DualSFT yields a more favorable joint-constrained trade-off than sequential hybrid baselines under matched budgets.

Related papers

• RIMS: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation [19.76481955873559]
  Small-scale language models (SLMs) are attractive for retrieval-augmented generation (RAG) in resource-constrained settings.<n>We propose RIMS, a three-stage preference optimization framework.<n>We show that our approach outperforms state-of-the-art baselines across multiple SLM backbones.
  arXiv  Detail & Related papers  (2026-07-17T18:24:19Z)
• Fixed-Parameter Tractability of Private Synthetic Data Generation [84.99100741615423]
  We study the problem of generating synthetic data under differential privacy.<n>Our algorithms attain optimal error rates across all regimes and are realized by two different approaches.
  arXiv  Detail & Related papers  (2026-06-09T15:14:11Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• Contextual Plackett-Luce: An Efficient Neural Model for Probabilistic Sequence Selection under Ambiguity [4.162695034027842]
  Contextual Plackett-Luce is a structured probabilistic model for sequence selection.<n>It can be viewed as a hybrid between fully autoregressive prediction and parallel sequence selection.<n>It achieves improved structural consistency and under ambiguous supervision compared to strong parallel baselines.
  arXiv  Detail & Related papers  (2026-05-09T18:52:53Z)
• Towards a Data-Parameter Correspondence for LLMs: A Preliminary Discussion [10.237445112597095]
  Large language model optimization has historically bifurcated into isolated data-centric and model-centric paradigms.<n>This paper establishes a unified emphdata- parameter correspondence, revealing these seemingly disparate operations as dual manifestations of the same geometric structure.
  arXiv  Detail & Related papers  (2026-04-19T11:18:03Z)
• GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry [4.94446914034065]
  GIST (Gradient Isometric Subspace Transformation) replaces axis-aligned scaling with robust subspace alignment.<n>We show that GIST matches or outperforms the state-of-the-art baseline with only 0.29% of the storage and 25% of the computational time.
  arXiv  Detail & Related papers  (2026-02-20T19:44:24Z)
• Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning [71.30276778807068]
  We propose a unified framework that strategically coordinates sample pruning and token pruning.<n>Q-Tuning achieves a +38% average improvement over the full-data SFT baseline using only 12.5% of the original training data.
  arXiv  Detail & Related papers  (2025-09-28T13:27:38Z)
• TACOS: Open Tagging and Comparative Scoring for Instruction Fine-Tuning Data Selection [9.020110377060153]
  We present TACOS, an innovative method that integrates Open Tagging and Comparative Scoring for IFT data selection.<n>To capture data diversity, we leverage LLMs to assign open-domain tags to human queries.<n>We suggest a comparative scoring method that allows the relative quality evaluation of samples within a cluster, avoiding inconsistent criteria seen in singleton-based evaluations.
  arXiv  Detail & Related papers  (2025-07-04T15:46:07Z)
• Adaptive and Robust DBSCAN with Multi-agent Reinforcement Learning [53.527506374566485]
  We propose a novel Adaptive and Robust DBSCAN with Multi-agent Reinforcement Learning cluster framework, namely AR-DBSCAN.<n>We show that AR-DBSCAN not only improves clustering accuracy by up to 144.1% and 175.3% in the NMI and ARI metrics, respectively, but also is capable of robustly finding dominant parameters.
  arXiv  Detail & Related papers  (2025-05-07T11:37:23Z)
• Joint Transmit and Pinching Beamforming for Pinching Antenna Systems (PASS): Optimization-Based or Learning-Based? [89.05848771674773]
  A novel antenna system ()-enabled downlink multi-user multiple-input single-output (MISO) framework is proposed.<n>It consists of multiple waveguides, which equip numerous low-cost antennas, named (PAs)<n>The positions of PAs can be reconfigured to both spanning large-scale path and space.
  arXiv  Detail & Related papers  (2025-02-12T18:54:10Z)
• Towards Generalizable Trajectory Prediction Using Dual-Level Representation Learning And Adaptive Prompting [107.4034346788744]
  Existing vehicle trajectory prediction models struggle with generalizability, prediction uncertainties, and handling complex interactions.<n>We propose Perceiver with Register queries (PerReg+), a novel trajectory prediction framework that introduces: (1) Dual-Level Representation Learning via Self-Distillation (SD) and Masked Reconstruction (MR), capturing global context and fine-grained details; (2) Enhanced Multimodality using register-based queries and pretraining, eliminating the need for clustering and suppression; and (3) Adaptive Prompt Tuning during fine-tuning, freezing the main architecture and optimizing a small number of prompts for efficient adaptation.
  arXiv  Detail & Related papers  (2025-01-08T20:11:09Z)
• Incorporating Arbitrary Matrix Group Equivariance into KANs [69.30866522377694]
  Kolmogorov-Arnold Networks (KANs) have seen great success in scientific domains.<n>We propose Equivariant Kolmogorov-Arnold Networks (EKAN) to broaden their applicability to more fields.
  arXiv  Detail & Related papers  (2024-10-01T06:34:58Z)
• FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneity [4.2059108111562935]
  Federated Learning (FL) is an emerging framework for distributed processing of large data volumes by edge devices. In this paper, we introduce a new FLAD FedADMM based protocol. We show that FedADMM consistently outperforms all baseline methods in terms of communication efficiency.
  arXiv  Detail & Related papers  (2022-04-07T15:58:33Z)
• Ensemble Model with Batch Spectral Regularization and Data Blending for Cross-Domain Few-Shot Learning with Unlabeled Data [75.94147344921355]
  We build a multi-branch ensemble framework by using diverse feature transformation matrices. We propose a data blending method to exploit the unlabeled data and augment the sparse support set in the target domain.
  arXiv  Detail & Related papers  (2020-06-08T02:27:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.