Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data

• URL: http://arxiv.org/abs/2502.04380v1
• Date: Wed, 05 Feb 2025 17:21:01 GMT
• Title: Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data
• Authors: Zhenqing Ling, Daoyuan Chen, Liuyi Yao, Yaliang Li, Ying Shen,
• Abstract summary: We study the role of data diversity in enhancing the overall abilities of large language models (LLMs)<n>We propose a new method that gives the LLM a dual identity: an output model to cognitively probe and select data based on diversity reward, as well as an input model to be tuned with the selected data.
• Score: 36.277423093218275
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-tuning large language models (LLMs) using diverse datasets is crucial for enhancing their overall performance across various domains. In practical scenarios, existing methods based on modeling the mixture proportions of data composition often struggle with data whose domain labels are missing, imprecise or non-normalized, while methods based on data selection usually encounter difficulties in balancing multi-domain performance. To address these challenges, in this paper, we study the role of data diversity in enhancing the overall abilities of LLMs by empirically constructing contrastive data pools and theoretically deriving explanations for both inter- and intra-diversity. Building upon the insights gained, we propose a new method that gives the LLM a dual identity: an output model to cognitively probe and select data based on diversity reward, as well as an input model to be tuned with the selected data. Extensive experiments show that the proposed method notably boosts performance across domain-undetermined data and a series of foundational downstream tasks when applied to various advanced LLMs. We release our code and hope this study can shed light on the understanding of data diversity and advance feedback-driven data-model co-development for LLMs.

Related papers

• MDIT: A Model-free Data Interpolation Method for Diverse Instruction Tuning [20.79390984800288]
  Large Language Models (LLMs) are increasingly applied across various tasks. We propose MDIT, a novel model-free data method for diverse instruction tuning. Extensive experiments show that our method achieves superior performance in multiple benchmark tasks.
  arXiv  Detail & Related papers  (2025-04-09T21:28:17Z)
• Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric [48.81957145701228]
  We propose a new diversity metric based on sample-level "novelty" We show that NovelSum accurately captures diversity variations and achieves a 0.97 correlation with instruction-tuned model performance.
  arXiv  Detail & Related papers  (2025-02-24T14:20:22Z)
• Unsupervised Topic Models are Data Mixers for Pre-training Language Models [6.77198566340415]
  We propose a topic-based data mixing strategy for large language models (LLMs) DataWeave employs a multi-stage clustering process to group semantically similar documents. We confirm that the topics Science and Relationships are particularly effective, yielding the most substantial performance improvements.
  arXiv  Detail & Related papers  (2025-02-24T03:25:56Z)
• Empowering Large Language Models in Wireless Communication: A Novel Dataset and Fine-Tuning Framework [81.29965270493238]
  We develop a specialized dataset aimed at enhancing the evaluation and fine-tuning of large language models (LLMs) for wireless communication applications. The dataset includes a diverse set of multi-hop questions, including true/false and multiple-choice types, spanning varying difficulty levels from easy to hard. We introduce a Pointwise V-Information (PVI) based fine-tuning method, providing a detailed theoretical analysis and justification for its use in quantifying the information content of training data.
  arXiv  Detail & Related papers  (2025-01-16T16:19:53Z)
• Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures [16.334964586540178]
  A large amount of instructional text data is essential to enhance the performance of large language models.<n>FedAMoLE is a lightweight personalized federated fine-tuning framework.
  arXiv  Detail & Related papers  (2024-11-28T13:20:38Z)
• FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data [64.50893177169996]
  Fine-tuning Multimodal Large Language Models (MLLMs) with Federated Learning (FL) allows for expanding the training data scope by including private data sources. We introduce a benchmark for evaluating various downstream tasks in the federated fine-tuning of MLLMs within multimodal heterogeneous scenarios. We develop a general FedMLLM framework that integrates four representative FL methods alongside two modality-agnostic strategies.
  arXiv  Detail & Related papers  (2024-11-22T04:09:23Z)
• On the Diversity of Synthetic Data and its Impact on Training Large Language Models [34.00031258223175]
  Large Language Models (LLMs) have accentuated the need for diverse, high-quality pre-training data. Synthetic data emerges as a viable solution to the challenges of data scarcity and inaccessibility. We study the downstream effects of synthetic data diversity during both the pre-training and fine-tuning stages.
  arXiv  Detail & Related papers  (2024-10-19T22:14:07Z)
• A Framework for Fine-Tuning LLMs using Heterogeneous Feedback [69.51729152929413]
  We present a framework for fine-tuning large language models (LLMs) using heterogeneous feedback. First, we combine the heterogeneous feedback data into a single supervision format, compatible with methods like SFT and RLHF. Next, given this unified feedback dataset, we extract a high-quality and diverse subset to obtain performance increases.
  arXiv  Detail & Related papers  (2024-08-05T23:20:32Z)
• Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning [64.5243480989869]
  coding data is known to boost reasoning abilities during pretraining. Its role in activating internal reasoning capacities during IFT remains understudied. This paper investigates how coding data impact LLMs' reasoning capacities during IFT stage.
  arXiv  Detail & Related papers  (2024-05-30T23:20:25Z)
• SilverSight: A Multi-Task Chinese Financial Large Language Model Based on Adaptive Semantic Space Learning [4.540505713937026]
  This study introduces an Adaptive Semantic Space Learning (ASSL) framework to enhance the performance and selection efficacy of multi-expert models. Our research findings demonstrate that our framework can achieve results close to those obtained with full data training using only 10% of the data, while also exhibiting strong generalization capabilities.
  arXiv  Detail & Related papers  (2024-04-07T13:02:21Z)
• Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges [47.45993726498343]
  Data augmentation (DA) has emerged as a pivotal technique for enhancing model performance by diversifying training examples without the need for additional data collection. This survey explores the transformative impact of large language models (LLMs) on DA, particularly addressing the unique challenges and opportunities they present in the context of natural language processing (NLP) and beyond.
  arXiv  Detail & Related papers  (2024-03-05T14:11:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.