Self-Instruct: Aligning Language Models with Self-Generated Instructions
- URL: http://arxiv.org/abs/2212.10560v2
- Date: Thu, 25 May 2023 23:50:07 GMT
- Title: Self-Instruct: Aligning Language Models with Self-Generated Instructions
- Authors: Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith,
Daniel Khashabi, Hannaneh Hajishirzi
- Abstract summary: Self-Instruct is a framework for improving the instruction-following capabilities of pretrained language models.
Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model.
For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin.
- Score: 76.42871502364697
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Large "instruction-tuned" language models (i.e., finetuned to respond to
instructions) have demonstrated a remarkable ability to generalize zero-shot to
new tasks. Nevertheless, they depend heavily on human-written instruction data
that is often limited in quantity, diversity, and creativity, therefore
hindering the generality of the tuned model. We introduce Self-Instruct, a
framework for improving the instruction-following capabilities of pretrained
language models by bootstrapping off their own generations. Our pipeline
generates instructions, input, and output samples from a language model, then
filters invalid or similar ones before using them to finetune the original
model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute
improvement over the original model on Super-NaturalInstructions, on par with
the performance of InstructGPT-001, which was trained with private user data
and human annotations. For further evaluation, we curate a set of
expert-written instructions for novel tasks, and show through human evaluation
that tuning GPT3 with Self-Instruct outperforms using existing public
instruction datasets by a large margin, leaving only a 5% absolute gap behind
InstructGPT-001. Self-Instruct provides an almost annotation-free method for
aligning pre-trained language models with instructions, and we release our
large synthetic dataset to facilitate future studies on instruction tuning. Our
code and data are available at https://github.com/yizhongw/self-instruct.
Related papers
- Privacy-Preserving Instructions for Aligning Large Language Models [45.712223061539284]
We propose using synthetic instructions to replace real instructions in data annotation and model fine-tuning.
Formal differential privacy is guaranteed by generating those synthetic instructions using privately fine-tuned generators.
In supervised fine-tuning, models trained with private synthetic instructions outperform leading open-source models such as Vicuna.
arXiv Detail & Related papers (2024-02-21T09:45:08Z) - Self-Alignment with Instruction Backtranslation [162.02529653768096]
We present a method to build a high quality instruction following language model by automatically labelling human-written text with corresponding instructions.
Our approach, named instruction backtranslation, starts with a language model finetuned on a small amount of seed data, and a given web corpus.
arXiv Detail & Related papers (2023-08-11T17:47:54Z) - Instruction-following Evaluation through Verbalizer Manipulation [64.73188776428799]
We propose a novel instruction-following evaluation protocol called verbalizer manipulation.
It instructs the model to verbalize the task label with words aligning with model priors to different extents.
We observe that the instruction-following abilities of models, across different families and scales, are significantly distinguished by their performance on less natural verbalizers.
arXiv Detail & Related papers (2023-07-20T03:54:24Z) - Evaluating the Zero-shot Robustness of Instruction-tuned Language Models [23.488398944358643]
We find that using novel (unobserved) but appropriate instruction phrasings consistently degrades model performance.
We propose a simple method to mitigate this issue by introducing soft prompt'' embedding parameters.
We show that this method consistently improves the robustness of instruction-tuned models.
arXiv Detail & Related papers (2023-06-20T03:48:51Z) - Unnatural Instructions: Tuning Language Models with (Almost) No Human
Labor [48.116843121810135]
We introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor.
We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth.
This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs.
arXiv Detail & Related papers (2022-12-19T18:21:00Z) - Instruction Induction: From Few Examples to Natural Language Task
Descriptions [55.139554327372934]
We show that language models can explicitly infer an underlying task from a few demonstrations by prompting them to generate a natural language instruction that fits the examples.
InstructGPT achieves 65.7% of human performance in our execution-based metric, while the original GPT-3 model reaches only 9.8% of human performance.
arXiv Detail & Related papers (2022-05-22T09:22:37Z) - Training language models to follow instructions with human feedback [29.590666996229206]
We show an avenue for aligning language models with user intent by fine-tuning with human feedback.
InstructGPT models show improvements in truthfulness and reductions in toxic output generation.
arXiv Detail & Related papers (2022-03-04T07:04:42Z) - Finetuned Language Models Are Zero-Shot Learners [67.70352207685558]
We show that instruction tuning boosts zero-shot performance on unseen tasks.
We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates.
We evaluate this instruction-tuned model, which we call FLAN, on unseen task types.
arXiv Detail & Related papers (2021-09-03T17:55:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.