Large Language Model Instruction Following: A Survey of Progresses and Challenges

• URL: http://arxiv.org/abs/2303.10475v8
• Date: Sat, 25 May 2024 03:21:21 GMT
• Title: Large Language Model Instruction Following: A Survey of Progresses and Challenges
• Authors: Renze Lou, Kai Zhang, Wenpeng Yin,
• Abstract summary: This paper tries to summarize and provide insights to the current research on instruction following. To our knowledge, this is the first comprehensive survey about instruction following.
• Score: 15.94137745420097
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Task semantics can be expressed by a set of input-output examples or a piece of textual instruction. Conventional machine learning approaches for natural language processing (NLP) mainly rely on the availability of large-scale sets of task-specific examples. Two issues arise: first, collecting task-specific labeled examples does not apply to scenarios where tasks may be too complicated or costly to annotate, or the system is required to handle a new task immediately; second, this is not user-friendly since end-users are probably more willing to provide task description rather than a set of examples before using the system. Therefore, the community is paying increasing interest in a new supervision-seeking paradigm for NLP: learning to follow task instructions, i.e., instruction following. Despite its impressive progress, there are some common issues that the community struggles with. This survey paper tries to summarize and provide insights to the current research on instruction following, particularly, by answering the following questions: (i) What is task instruction, and what instruction types exist? (ii) How to model instructions? (iii) What are popular instruction following datasets and evaluation metrics? (iv) What factors influence and explain the instructions' performance? (v) What challenges remain in instruction following? To our knowledge, this is the first comprehensive survey about instruction following.

Related papers

• Instruction Embedding: Latent Representations of Instructions Towards Task Identification [20.327984896070053]
  For instructional data, the most important aspect is the task it represents, rather than the specific semantics and knowledge information. In this work, we introduce a new concept, instruction embedding, and construct Instruction Embedding Benchmark (IEB) for its training and evaluation.
  arXiv  Detail & Related papers  (2024-09-29T12:12:24Z)
• The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models [48.455388608863785]
  We introduce a benchmark designed to evaluate models' abilities to follow multiple instructions through sequential instruction following tasks. Our benchmark evaluates instruction following using four tasks (text modification, question answering, mathematics, and security rules) More recent and larger models significantly outperform their older and smaller counterparts on the SIFo tasks, validating the benchmark's effectiveness.
  arXiv  Detail & Related papers  (2024-06-28T15:34:26Z)
• An Incomplete Loop: Deductive, Inductive, and Abductive Learning in Large Language Models [99.31449616860291]
  Modern language models (LMs) can learn to perform new tasks in different ways. In instruction following, the target task is described explicitly in natural language; in few-shot prompting, the task is specified implicitly. In instruction inference, LMs are presented with in-context examples and are then prompted to generate a natural language task description.
  arXiv  Detail & Related papers  (2024-04-03T19:31:56Z)
• LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity [11.44413929033824]
  We develop LINGO, a novel visual analytics interface that supports an effective, task-driven workflow. We conduct a user study with both novice and expert instruction creators, over a dataset of 1,616 linguistic tasks and their natural language instructions. For both user groups, LINGO promotes the creation of more difficult tasks for pre-trained models, that contain higher linguistic diversity and lower instruction bias.
  arXiv  Detail & Related papers  (2023-04-12T22:55:52Z)
• Robustness of Learning from Task Instructions [15.462970803323563]
  Traditional supervised learning mostly works on individual tasks and requires training on a large set of task-specific examples. To build a system that can quickly and easily generalize to new tasks, task instructions have been adopted as an emerging trend of supervision. This work investigates the system robustness when the instructions of new tasks are (i) manipulated, (ii) paraphrased, or (iii) from different levels of conciseness.
  arXiv  Detail & Related papers  (2022-12-07T17:54:59Z)
• Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
  Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
  arXiv  Detail & Related papers  (2022-10-12T15:02:04Z)
• ConTinTin: Continual Learning from Task Instructions [101.36836925135091]
  This work defines a new learning paradigm ConTinTin, in which a system should learn a sequence of new tasks one by one, each task is explained by a piece of textual instruction. To our knowledge, this is the first time to study ConTinTin in NLP.
  arXiv  Detail & Related papers  (2022-03-16T10:27:18Z)
• Instance-aware Prompt Learning for Language Understanding and Generation [49.22899822734549]
  We propose an instance-aware prompt learning method that learns a different prompt for each instance. Our method achieves the state-of-the-art on the SuperGLUE few-shot learning benchmark.
  arXiv  Detail & Related papers  (2022-01-18T17:03:25Z)
• CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems [56.302581679816775]
  This paper proposes Comprehensive Instruction (CINS) that exploits PLMs with task-specific instructions. We design a schema (definition, constraint, prompt) of instructions and their customized realizations for three important downstream tasks in ToD. Experiments are conducted on these ToD tasks in realistic few-shot learning scenarios with small validation data.
  arXiv  Detail & Related papers  (2021-09-10T03:23:06Z)
• Visual-and-Language Navigation: A Survey and Taxonomy [1.0742675209112622]
  This paper provides a comprehensive survey on Visual-and-Language Navigation (VLN) tasks. According to when the instructions are given, the tasks can be divided into single-turn and multi-turn. This taxonomy enable researchers to better grasp the key point of a specific task and identify directions for future research.
  arXiv  Detail & Related papers  (2021-08-26T01:51:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.