Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptation

• URL: http://arxiv.org/abs/2602.06359v1
• Date: Fri, 06 Feb 2026 03:41:40 GMT
• Title: Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptation
• Authors: Xiyang Zhang, Yuanhe Tian, Hongzhi Wang, Yan Song,
• Abstract summary: Fine-tuning large language models for specialized domains often necessitates a trade-off between acquiring domain expertise and retaining general reasoning capabilities.<n>We propose Orthogonal Gradient Selection (OGS), a data-centric method that harmonizes domain performance, general capability retention, and training efficiency.
• Score: 21.694351921779845
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-tuning large language models (LLMs) for specialized domains often necessitates a trade-off between acquiring domain expertise and retaining general reasoning capabilities, a phenomenon known as catastrophic forgetting. Existing remedies face a dichotomy: gradient surgery methods offer geometric safety but incur prohibitive computational costs via online projections, while efficient data selection approaches reduce overhead but remain blind to conflict-inducing gradient directions. In this paper, we propose Orthogonal Gradient Selection (OGS), a data-centric method that harmonizes domain performance, general capability retention, and training efficiency. OGS shifts the geometric insights of gradient projection from the optimizer to the data selection stage by treating data selection as a constrained decision-making process. By leveraging a lightweight Navigator model and reinforcement learning techniques, OGS dynamically identifies training samples whose gradients are orthogonal to a general-knowledge anchor. This approach ensures naturally safe updates for target models without modifying the optimizer or incurring runtime projection costs. Experiments across medical, legal, and financial domains demonstrate that OGS achieves excellent results, significantly improving domain performance and training efficiency while maintaining or even enhancing performance on general tasks such as GSM8K.

Related papers

• Closing the Train-Test Gap in World Models for Gradient-Based Planning [64.36544881136405]
  We propose improved methods for training world models that enable efficient gradient-based planning.<n>At test time, our approach outperforms or matches the classical gradient-free cross-entropy method.
  arXiv  Detail & Related papers  (2025-12-10T18:59:45Z)
• Feature-Space Planes Searcher: A Universal Domain Adaptation Framework for Interpretability and Computational Efficiency [7.889121135601528]
  Current unsupervised domain adaptation methods rely on fine-tuning feature extractors.<n>We propose Feature-space Planes Searcher (FPS) as a novel domain adaptation framework.<n>We show that FPS achieves competitive or superior performance to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-08-26T05:39:21Z)
• Hindsight-Guided Momentum (HGM) Optimizer: An Approach to Adaptive Learning Rate [0.0]
  We introduce Hindsight-Guided Momentum, a first-order optimization algorithm that adaptively scales learning rates based on recent updates.<n>HGM addresses this by a hindsight mechanism that accelerates the learning rate between coherent and conflicting directions.
  arXiv  Detail & Related papers  (2025-06-22T08:02:19Z)
• Graph Data Selection for Domain Adaptation: A Model-Free Approach [54.27731120381295]
  Graph domain adaptation (GDA) is a fundamental task in graph machine learning.<n>We propose a novel model-free framework, GRADATE, that selects the best training data from the source domain for the classification task on the target domain.<n>We show GRADATE outperforms existing selection methods and enhances off-the-shelf GDA methods with much fewer training data.
  arXiv  Detail & Related papers  (2025-05-22T21:18:39Z)
• Efficient Distributed Optimization under Heavy-Tailed Noise [32.96984712007111]
  TailOPT is designed to address heavy-tailed noise with potentially gradient variance and local updates.<n>$Bi2Clip$ performs coordinate-wise clipping at both the inner and outers, achieving adaptive-like performance.<n>$Bi2Clip$ demonstrates superior performance on several language tasks and models, outperforming state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-02-06T15:47:18Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Taxonomy Adaptive Cross-Domain Adaptation in Medical Imaging via Optimization Trajectory Distillation [73.83178465971552]
  The success of automated medical image analysis depends on large-scale and expert-annotated training sets. Unsupervised domain adaptation (UDA) has been raised as a promising approach to alleviate the burden of labeled data collection. We propose optimization trajectory distillation, a unified approach to address the two technical challenges from a new perspective.
  arXiv  Detail & Related papers  (2023-07-27T08:58:05Z)
• Cluster-level pseudo-labelling for source-free cross-domain facial expression recognition [94.56304526014875]
  We propose the first Source-Free Unsupervised Domain Adaptation (SFUDA) method for Facial Expression Recognition (FER) Our method exploits self-supervised pretraining to learn good feature representations from the target data. We validate the effectiveness of our method in four adaptation setups, proving that it consistently outperforms existing SFUDA methods when applied to FER.
  arXiv  Detail & Related papers  (2022-10-11T08:24:50Z)
• Data Augmentation through Expert-guided Symmetry Detection to Improve Performance in Offline Reinforcement Learning [0.0]
  offline estimation of the dynamical model of a Markov Decision Process (MDP) is a non-trivial task. Recent works showed that an expert-guided pipeline relying on Density Estimation methods effectively detects this structure in deterministic environments. We show that the former results lead to a performance improvement when solving the learned MDP and then applying the optimized policy in the real environment.
  arXiv  Detail & Related papers  (2021-12-18T14:32:32Z)
• Domain Adaptive Person Re-Identification via Coupling Optimization [58.567492812339566]
  Domain adaptive person Re-Identification (ReID) is challenging owing to the domain gap and shortage of annotations on target scenarios. This paper proposes a coupling optimization method including the Domain-Invariant Mapping (DIM) method and the Global-Local distance Optimization ( GLO) GLO is designed to train the ReID model with unsupervised setting on the target domain.
  arXiv  Detail & Related papers  (2020-11-06T14:01:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.