Robustness of Learning from Task Instructions
- URL: http://arxiv.org/abs/2212.03813v4
- Date: Tue, 23 May 2023 13:39:39 GMT
- Title: Robustness of Learning from Task Instructions
- Authors: Jiasheng Gu, Hongyu Zhao, Hanzi Xu, Liangyu Nie, Hongyuan Mei and
Wenpeng Yin
- Abstract summary: Traditional supervised learning mostly works on individual tasks and requires training on a large set of task-specific examples.
To build a system that can quickly and easily generalize to new tasks, task instructions have been adopted as an emerging trend of supervision.
This work investigates the system robustness when the instructions of new tasks are (i) manipulated, (ii) paraphrased, or (iii) from different levels of conciseness.
- Score: 15.462970803323563
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Traditional supervised learning mostly works on individual tasks and requires
training on a large set of task-specific examples. This paradigm seriously
hinders the development of task generalization since preparing a task-specific
example set is costly. To build a system that can quickly and easily generalize
to new tasks, task instructions have been adopted as an emerging trend of
supervision recently. These instructions give the model the definition of the
task and allow the model to output the appropriate answer based on the
instructions and inputs. However, task instructions are often expressed in
different forms, which can be interpreted from two threads: first, some
instructions are short sentences and are pretrained language model (PLM)
oriented, such as prompts, while other instructions are paragraphs and are
human-oriented, such as those in Amazon MTurk; second, different end-users very
likely explain the same task with instructions of different textual
expressions. A robust system for task generalization should be able to handle
any new tasks regardless of the variability of instructions.
However, the system robustness in dealing with instruction-driven task
generalization is still unexplored. This work investigates the system
robustness when the instructions of new tasks are (i) manipulated, (ii)
paraphrased, or (iii) from different levels of conciseness. To our knowledge,
this is the first work that systematically studies how robust a PLM is when it
is supervised by instructions with different factors of variability.
Related papers
- An Incomplete Loop: Deductive, Inductive, and Abductive Learning in Large Language Models [99.31449616860291]
Modern language models (LMs) can learn to perform new tasks in different ways.
In instruction following, the target task is described explicitly in natural language; in few-shot prompting, the task is specified implicitly.
In instruction inference, LMs are presented with in-context examples and are then prompted to generate a natural language task description.
arXiv Detail & Related papers (2024-04-03T19:31:56Z) - Instruction Diversity Drives Generalization To Unseen Tasks [1.9059113568275998]
Generalization emerges once a diverse enough set of tasks is provided, even though very few examples are provided for each task.
Generalization emerges once a diverse enough set of tasks is provided, even though very few examples are provided for each task.
arXiv Detail & Related papers (2024-02-16T18:47:21Z) - Active Instruction Tuning: Improving Cross-Task Generalization by
Training on Prompt Sensitive Tasks [101.40633115037983]
Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models (LLMs) on a massive amount of diverse tasks with instructions.
How to select new tasks to improve the performance and generalizability of IT models remains an open question.
We propose active instruction tuning based on prompt uncertainty, a novel framework to identify informative tasks, and then actively tune the models on the selected tasks.
arXiv Detail & Related papers (2023-11-01T04:40:05Z) - Did You Read the Instructions? Rethinking the Effectiveness of Task
Definitions in Instruction Learning [74.70157466822612]
We systematically study the role of task definitions in instruction learning.
We find that model performance drops substantially when removing contents describing the task output.
We propose two strategies to help models better leverage task instructions.
arXiv Detail & Related papers (2023-06-01T21:11:24Z) - LINGO : Visually Debiasing Natural Language Instructions to Support Task
Diversity [11.44413929033824]
We develop LINGO, a novel visual analytics interface that supports an effective, task-driven workflow.
We conduct a user study with both novice and expert instruction creators, over a dataset of 1,616 linguistic tasks and their natural language instructions.
For both user groups, LINGO promotes the creation of more difficult tasks for pre-trained models, that contain higher linguistic diversity and lower instruction bias.
arXiv Detail & Related papers (2023-04-12T22:55:52Z) - Large Language Model Instruction Following: A Survey of Progresses and Challenges [15.94137745420097]
This paper tries to summarize and provide insights to the current research on instruction following.
To our knowledge, this is the first comprehensive survey about instruction following.
arXiv Detail & Related papers (2023-03-18T19:17:47Z) - Using Both Demonstrations and Language Instructions to Efficiently Learn
Robotic Tasks [21.65346551790888]
DeL-TaCo is a method for conditioning a robotic policy on task embeddings comprised of two components: a visual demonstration and a language instruction.
To our knowledge, this is the first work to show that simultaneously conditioning a multi-task robotic manipulation policy on both demonstration and language embeddings improves sample efficiency and generalization over conditioning on either modality alone.
arXiv Detail & Related papers (2022-10-10T08:06:58Z) - Improving Task Generalization via Unified Schema Prompt [87.31158568180514]
Unified Prompt is a flexible and prompting method, which automatically customizes the learnable prompts for each task according to the task input schema.
It models the shared knowledge between tasks, while keeping the characteristics of different task schema.
The framework achieves strong zero-shot and few-shot performance on 16 unseen tasks downstream from 8 task types.
arXiv Detail & Related papers (2022-08-05T15:26:36Z) - Fast Inference and Transfer of Compositional Task Structures for
Few-shot Task Generalization [101.72755769194677]
We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph.
Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks.
Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
arXiv Detail & Related papers (2022-05-25T10:44:25Z) - ConTinTin: Continual Learning from Task Instructions [101.36836925135091]
This work defines a new learning paradigm ConTinTin, in which a system should learn a sequence of new tasks one by one, each task is explained by a piece of textual instruction.
To our knowledge, this is the first time to study ConTinTin in NLP.
arXiv Detail & Related papers (2022-03-16T10:27:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.