Efficient Response Generation Method Selection for Fine-Tuning Large Language Models

• URL: http://arxiv.org/abs/2502.11779v1
• Date: Mon, 17 Feb 2025 13:14:11 GMT
• Title: Efficient Response Generation Method Selection for Fine-Tuning Large Language Models
• Authors: Xuan Ren, Qi Chen, Lingqiao Liu,
• Abstract summary: Recent studies have observed that the choice of output variation used in training can affect the model's performance. This paper proposes a scalable, approximate method for estimating the quality of a small subset of generated training data. We show that an LLM trained on data generated by the selected strategy could lead to a significant performance gain.
• Score: 28.717420152590204
• License:
• Abstract: The training data for fine-tuning large language models (LLMs) is typically structured as input-output pairs. However, for many tasks, there can be multiple equally valid output variations for the same input. Recent studies have observed that the choice of output variation used in training can affect the model's performance. This raises an important question: how can we generate the most effective output from the many possible response generation strategy options? Rather than relying on the traditional but resource-intensive train-and-evaluate approach, this paper proposes a scalable, approximate method for estimating the quality of a small subset of generated training data derived from the same input. We then evaluate how well this small subset of generated output fits the target model we are trying to train. We present a large-scale benchmark covering diverse reasoning-based datasets to support our study. The central idea is that a good output should closely resemble the output generated by the target LLM. We formalize this 'closeness' as the expected alignment score between a candidate output and the output sampled from the target LLM. We connect this measurement to the perplexity metric used in previous literature and demonstrate that leveraging an alignment-based metric can provide better predictions of model performance. Using this strategy, we can evaluate a small subset of the generated output from each response generation strategy option, then select the most effective strategy. We show that an LLM trained on data generated by the selected strategy could lead to a significant performance gain in many cases.

Related papers

• Scalable Influence and Fact Tracing for Large Language Model Pretraining [14.598556308631018]
  Training data attribution (TDA) methods aim to attribute model outputs back to specific training examples. We refine existing gradient-based methods to work effectively at scale. We release our prompt set and model outputs, along with a web-based visualization tool to explore influential examples.
  arXiv  Detail & Related papers  (2024-10-22T20:39:21Z)
• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• Graph-Structured Speculative Decoding [52.94367724136063]
  Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models. We introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. We observe a remarkable speedup of 1.73$times$ to 1.96$times$, significantly surpassing standard speculative decoding.
  arXiv  Detail & Related papers  (2024-07-23T06:21:24Z)
• LLM-Select: Feature Selection with Large Language Models [64.5099482021597]
  Large language models (LLMs) are capable of selecting the most predictive features, with performance rivaling the standard tools of data science. Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place.
  arXiv  Detail & Related papers  (2024-07-02T22:23:40Z)
• Aligning Large Language Models via Fine-grained Supervision [20.35000061196631]
  Pre-trained large-scale language models (LLMs) excel at producing coherent articles, yet their outputs may be untruthful, toxic, or fail to align with user expectations. Current approaches focus on using reinforcement learning with human feedback to improve model alignment. We propose a method to enhance LLM alignment through fine-grained token-level supervision.
  arXiv  Detail & Related papers  (2024-06-04T20:21:45Z)
• Regression-aware Inference with LLMs [52.764328080398805]
  We show that an inference strategy can be sub-optimal for common regression and scoring evaluation metrics. We propose alternate inference strategies that estimate the Bayes-optimal solution for regression and scoring metrics in closed-form from sampled responses.
  arXiv  Detail & Related papers  (2024-03-07T03:24:34Z)
• LLM-augmented Preference Learning from Natural Language [19.700169351688768]
  Large Language Models (LLMs) are equipped to deal with larger context lengths. LLMs can consistently outperform the SotA when the target text is large. Few-shot learning yields better performance than zero-shot learning.
  arXiv  Detail & Related papers  (2023-10-12T17:17:27Z)
• Enhancing Subtask Performance of Multi-modal Large Language Model [12.033301861738952]
  Multi-modal Large Language Model (MLLM) refers to a model expanded from a Large Language Model (LLM) that possesses the capability to handle and infer multi-modal data. This study selects multiple pre-trained models focused on the same subtask based on distinct evaluation approaches. The results from multiple pre-trained models for the same subtask are compared using the LLM, and the best result is chosen as the outcome for that subtask.
  arXiv  Detail & Related papers  (2023-08-31T05:37:21Z)
• Language models are weak learners [71.33837923104808]
  We show that prompt-based large language models can operate effectively as weak learners. We incorporate these models into a boosting approach, which can leverage the knowledge within the model to outperform traditional tree-based boosting. Results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning pipelines.
  arXiv  Detail & Related papers  (2023-06-25T02:39:19Z)
• DORE: Document Ordered Relation Extraction based on Generative Framework [56.537386636819626]
  This paper investigates the root cause of the underwhelming performance of the existing generative DocRE models. We propose to generate a symbolic and ordered sequence from the relation matrix which is deterministic and easier for model to learn. Experimental results on four datasets show that our proposed method can improve the performance of the generative DocRE models.
  arXiv  Detail & Related papers  (2022-10-28T11:18:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.