Ensemble-Instruct: Generating Instruction-Tuning Data with a Heterogeneous Mixture of LMs

• URL: http://arxiv.org/abs/2310.13961v1
• Date: Sat, 21 Oct 2023 10:21:17 GMT
• Title: Ensemble-Instruct: Generating Instruction-Tuning Data with a Heterogeneous Mixture of LMs
• Authors: Young-Suk Lee, Md Arafat Sultan, Yousef El-Kurdi, Tahira Naseem Asim Munawar, Radu Florian, Salim Roukos, Ram\'on Fernandez Astudillo
• Abstract summary: In-context learning (ICL) techniques can train strong conversational agents with only a small amount of human supervision. Here we explore the application of such techniques to language models that are much smaller (around 10B--40B parameters) and have permissive licenses. We find the Self-Instruct approach to be less effective at these sizes and propose new ICL methods that draw on two main ideas.
• Score: 23.38507910115345
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Using in-context learning (ICL) for data generation, techniques such as Self-Instruct (Wang et al., 2023) or the follow-up Alpaca (Taori et al., 2023) can train strong conversational agents with only a small amount of human supervision. One limitation of these approaches is that they resort to very large language models (around 175B parameters) that are also proprietary and non-public. Here we explore the application of such techniques to language models that are much smaller (around 10B--40B parameters) and have permissive licenses. We find the Self-Instruct approach to be less effective at these sizes and propose new ICL methods that draw on two main ideas: (a) Categorization and simplification of the ICL templates to make prompt learning easier for the LM, and (b) Ensembling over multiple LM outputs to help select high-quality synthetic examples. Our algorithm leverages the 175 Self-Instruct seed tasks and employs separate pipelines for instructions that require an input and instructions that do not. Empirical investigations with different LMs show that: (1) Our proposed method yields higher-quality instruction tuning data than Self-Instruct, (2) It improves performances of both vanilla and instruction-tuned LMs by significant margins, and (3) Smaller instruction-tuned LMs generate more useful outputs than their larger un-tuned counterparts. Our codebase is available at https://github.com/IBM/ensemble-instruct.

Related papers

• Align$^2$LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation [56.75665429851673]
  This paper introduces a novel instruction curation algorithm, derived from two unique perspectives, human and LLM preference alignment. Experiments demonstrate that we can maintain or even improve model performance by compressing synthetic multimodal instructions by up to 90%.
  arXiv  Detail & Related papers  (2024-09-27T08:20:59Z)
• Phased Instruction Fine-Tuning for Large Language Models [12.037895935630882]
  Phased Instruction Fine-Tuning (Phased IFT) is proposed, based on the idea that learning to follow instructions is a gradual process. It assesses instruction difficulty using GPT-4, divides the instruction data into subsets of increasing difficulty, and uptrains the model sequentially on these subsets. Experiments with Llama-2 7B/13B/70B, Llama3 8/70B and Mistral-7B models using Alpaca data show that Phased IFT significantly outperforms One-off IFT.
  arXiv  Detail & Related papers  (2024-06-01T04:25:26Z)
• Instruction Tuning With Loss Over Instructions [42.9106826952674]
  Instruction Modelling (IM) trains LMs by applying a loss function to the instruction and prompt part rather than solely to the output part. We show that, in many scenarios, IM can effectively improve the LM performance on both NLP tasks and open-ended generation benchmarks. Remarkably, in the most advantageous case, IM boosts model performance on AlpacaEval 1.0 by over 100%.
  arXiv  Detail & Related papers  (2024-05-23T10:12:03Z)
• Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought [51.240387516059535]
  We introduce a novel framework, LM-Guided CoT, that leverages a lightweight (i.e., 1B) language model (LM) for guiding a black-box large (i.e., >10B) LM in reasoning tasks. We optimize the model through 1) knowledge distillation and 2) reinforcement learning from rationale-oriented and task-oriented reward signals.
  arXiv  Detail & Related papers  (2024-04-04T12:46:37Z)
• Instruction Distillation Makes Large Language Models Efficient Zero-shot Rankers [56.12593882838412]
  We introduce a novel instruction distillation method to rank documents. We first rank documents using the effective pairwise approach with complex instructions, and then distill the teacher predictions to the pointwise approach with simpler instructions. Our approach surpasses the performance of existing supervised methods like monoT5 and is on par with the state-of-the-art zero-shot methods.
  arXiv  Detail & Related papers  (2023-11-02T19:16:21Z)
• BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing [10.698756010878688]
  We created the BioInstruct, comprising 25,005 instructions to instruction-tune large language models (LLMs) The instructions were created by prompting the GPT-4 language model with three-seed samples randomly drawn from an 80 human curated instructions. We evaluated these instruction-tuned LLMs on several BioNLP tasks, which can be grouped into three major categories: question answering(QA), information extraction(IE), and text generation(GEN)
  arXiv  Detail & Related papers  (2023-10-30T19:38:50Z)
• Tuna: Instruction Tuning using Feedback from Large Language Models [74.04950416204551]
  We propose finetuning an instruction-tuned large language model using our novel textitprobabilistic ranking and textitcontextual ranking approaches. Probabilistic ranking enables the instruction-tuned model to inherit the relative rankings of high-quality and low-quality responses from the teacher LLM. On the other hand, learning with contextual ranking allows the model to refine its own response distribution using the contextual understanding ability of stronger LLMs.
  arXiv  Detail & Related papers  (2023-10-20T09:55:06Z)
• Language models are weak learners [71.33837923104808]
  We show that prompt-based large language models can operate effectively as weak learners. We incorporate these models into a boosting approach, which can leverage the knowledge within the model to outperform traditional tree-based boosting. Results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning pipelines.
  arXiv  Detail & Related papers  (2023-06-25T02:39:19Z)
• Preference-grounded Token-level Guidance for Language Model Fine-tuning [105.88789610320426]
  Aligning language models with preferences is an important problem in natural language generation. For LM training, based on the amount of supervised data, we present two *minimalist* learning objectives that utilize the learned guidance. In experiments, our method performs competitively on two distinct representative LM tasks.
  arXiv  Detail & Related papers  (2023-06-01T07:00:07Z)
• LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions [28.937552799649808]
  Large language models (LLMs) with instruction fine-tuning demonstrate superior generative capabilities. We develop a large set of 2.58M instructions based on both existing and newly-generated instructions. We fine-tune a diverse herd of models, collectively referred to as LaMini-LM, which includes models from both the encoder-decoder and decoder-only families.
  arXiv  Detail & Related papers  (2023-04-27T17:58:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.