Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning

• URL: http://arxiv.org/abs/2306.01150v1
• Date: Thu, 1 Jun 2023 21:11:24 GMT
• Title: Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning
• Authors: Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, Chien-Sheng Jason Wu
• Abstract summary: We systematically study the role of task definitions in instruction learning. We find that model performance drops substantially when removing contents describing the task output. We propose two strategies to help models better leverage task instructions.
• Score: 74.70157466822612
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models (LLMs) have shown impressive performance in following natural language instructions to solve unseen tasks. However, it remains unclear whether models truly understand task definitions and whether the human-written definitions are optimal. In this paper, we systematically study the role of task definitions in instruction learning. We first conduct an ablation analysis informed by human annotations to understand which parts of a task definition are most important, and find that model performance only drops substantially when removing contents describing the task output, in particular label information. Next, we propose an automatic algorithm to compress task definitions to a minimal supporting set of tokens, and find that 60\% of tokens can be removed while maintaining or even improving model performance. Based on these results, we propose two strategies to help models better leverage task instructions: (1) providing only key information for tasks in a common structured format, and (2) adding a meta-tuning stage to help the model better understand the definitions. With these two strategies, we achieve a 4.2 Rouge-L improvement over 119 unseen test tasks.

Related papers

• Evaluating the Instruction-following Abilities of Language Models using Knowledge Tasks [4.945902994386117]
  We focus on developing a benchmark for instruction-following where it is easy to verify both task performance as well as instruction-following capabilities. We adapt existing knowledge benchmarks and augment them with instructions that are a) conditional on correctly answering the knowledge task or b) use the space of candidate options in multiple-choice knowledge-answering tasks. We find that even large-scale instruction-tuned LLMs fail to follow simple instructions in zero-shot settings.
  arXiv  Detail & Related papers  (2024-10-16T19:07:37Z)
• Automated Few-shot Classification with Instruction-Finetuned Language Models [76.69064714392165]
  We show that AuT-Few outperforms state-of-the-art few-shot learning methods. We also show that AuT-Few is the best ranking method across datasets on the RAFT few-shot benchmark.
  arXiv  Detail & Related papers  (2023-05-21T21:50:27Z)
• LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity [11.44413929033824]
  We develop LINGO, a novel visual analytics interface that supports an effective, task-driven workflow. We conduct a user study with both novice and expert instruction creators, over a dataset of 1,616 linguistic tasks and their natural language instructions. For both user groups, LINGO promotes the creation of more difficult tasks for pre-trained models, that contain higher linguistic diversity and lower instruction bias.
  arXiv  Detail & Related papers  (2023-04-12T22:55:52Z)
• Unsupervised Task Graph Generation from Instructional Video Transcripts [53.54435048879365]
  We consider a setting where text transcripts of instructional videos performing a real-world activity are provided. The goal is to identify the key steps relevant to the task as well as the dependency relationship between these key steps. We propose a novel task graph generation approach that combines the reasoning capabilities of instruction-tuned language models along with clustering and ranking components.
  arXiv  Detail & Related papers  (2023-02-17T22:50:08Z)
• Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding [51.31622274823167]
  We propose a hierarchical framework with a coarse-to-fine paradigm, with the bottom level shared to all the tasks, the mid-level divided to different groups, and the top-level assigned to each of the tasks. This allows our model to learn basic language properties from all tasks, boost performance on relevant tasks, and reduce the negative impact from irrelevant tasks.
  arXiv  Detail & Related papers  (2022-08-19T02:46:20Z)
• DeepStruct: Pretraining of Language Models for Structure Prediction [64.84144849119554]
  We pretrain language models on a collection of task-agnostic corpora to generate structures from text. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We show that a 10B parameter language model transfers non-trivially to most tasks and obtains state-of-the-art performance on 21 of 28 datasets.
  arXiv  Detail & Related papers  (2022-05-21T00:58:22Z)
• InstructionNER: A Multi-Task Instruction-Based Generative Framework for Few-shot NER [31.32381919473188]
  We propose a multi-task instruction-based generative framework, named InstructionNER, for low-resource named entity recognition. Specifically, we reformulate the NER task as a generation problem, which enriches source sentences with task-specific instructions and answer options, then inferences the entities and types in natural language. Experimental results show that our method consistently outperforms other baselines on five datasets in few-shot settings.
  arXiv  Detail & Related papers  (2022-03-08T07:56:36Z)
• Learning to Follow Language Instructions with Compositional Policies [22.778677208048475]
  We propose a framework that learns to execute natural language instructions in an environment consisting of goal-reaching tasks. We train a reinforcement learning agent to learn value functions that can be subsequently composed through a Boolean algebra. We fine-tune a seq2seq model pretrained on web-scale corpora to map language to logical expressions.
  arXiv  Detail & Related papers  (2021-10-09T21:28:26Z)
• Pre-training Text Representations as Meta Learning [113.3361289756749]
  We introduce a learning algorithm which directly optimize model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps.
  arXiv  Detail & Related papers  (2020-04-12T09:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.