Mitigating the Problem of Strong Priors in LMs with Context
Extrapolation
- URL: http://arxiv.org/abs/2401.17692v1
- Date: Wed, 31 Jan 2024 09:28:06 GMT
- Title: Mitigating the Problem of Strong Priors in LMs with Context
Extrapolation
- Authors: Raymond Douglas, Andis Draguns, Tom\'a\v{s} Gaven\v{c}iak
- Abstract summary: We develop a new technique for mitigating the problem of strong priors.
We take the original set of instructions, produce a weakened version of the original prompt, and extrapolate the continuation away from the weakened prompt.
This lets us infer how the model would continue a hypothetical strengthened set of instructions.
- Score: 0.6629765271909505
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Language models (LMs) have become important tools in a variety of
applications, from data processing to the creation of instruction-following
assistants. But despite their advantages, LMs have certain idiosyncratic
limitations such as the problem of `strong priors', where a model learns to
output typical continuations in response to certain, usually local, portions of
the input regardless of any earlier instructions. For example, prompt injection
attacks can induce models to ignore explicit directives. In some cases, larger
models have been shown to be more susceptible to these problems than similar
smaller models, an example of the phenomenon of `inverse scaling'. We develop a
new technique for mitigating the problem of strong priors: we take the original
set of instructions, produce a weakened version of the original prompt that is
even more susceptible to the strong priors problem, and then extrapolate the
continuation away from the weakened prompt. This lets us infer how the model
would continue a hypothetical strengthened set of instructions. Our technique
conceptualises LMs as mixture models which combine a family of data generation
processes, reinforcing the desired elements of the mixture. Our approach works
at inference time, removing any need for retraining. We apply it to eleven
models including GPT-2, GPT-3, Llama 2, and Mistral on four tasks, and find
improvements in 41/44. Across all 44 combinations the median increase in
proportion of tasks completed is 40%.
Related papers
- Recursive Introspection: Teaching Language Model Agents How to Self-Improve [30.086494067593268]
We develop RISE: Recursive IntroSpEction, an approach for fine-tuning large language models.
Our experiments show that RISE enables Llama2, Llama3, and Mistral models to improve themselves with more turns on math reasoning tasks.
arXiv Detail & Related papers (2024-07-25T17:35:59Z) - Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction [75.25114727856861]
Large language models (LLMs) tend to suffer from deterioration at the latter stage ofSupervised fine-tuning process.
We introduce a simple disperse-then-merge framework to address the issue.
Our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks.
arXiv Detail & Related papers (2024-05-22T08:18:19Z) - Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models [115.501751261878]
Fine-tuning language models(LMs) on human-generated data remains a prevalent practice.
We investigate whether we can go beyond human data on tasks where we have access to scalar feedback.
We find that ReST$EM$ scales favorably with model size and significantly surpasses fine-tuning only on human data.
arXiv Detail & Related papers (2023-12-11T18:17:43Z) - First-Step Advantage: Importance of Starting Right in Multi-Step Math Reasoning [11.75364271481855]
Language models can solve complex reasoning tasks better by learning to generate rationales for their predictions.
We observe that smaller models in particular when corrected, can solve a task that they would have otherwise struggled with.
We propose QuestCoT, where a smaller model first asks itself how to start, before proceeding with a chain of reasoning.
arXiv Detail & Related papers (2023-11-14T06:45:31Z) - AdaMerging: Adaptive Model Merging for Multi-Task Learning [68.75885518081357]
This paper introduces an innovative technique called Adaptive Model Merging (AdaMerging)
It aims to autonomously learn the coefficients for model merging, either in a task-wise or layer-wise manner, without relying on the original training data.
Compared to the current state-of-the-art task arithmetic merging scheme, AdaMerging showcases a remarkable 11% improvement in performance.
arXiv Detail & Related papers (2023-10-04T04:26:33Z) - Inverse Scaling: When Bigger Isn't Better [80.42834197416444]
Large language models (LMs) show predictable improvements to overall loss with increased scale.
We present evidence for the claim that LMs may show inverse scaling, or worse task performance with increased scale.
arXiv Detail & Related papers (2023-06-15T20:11:23Z) - Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large
Language Models [11.57282859281814]
We consider different knowledge levels and attribution strategies, and find that we can correctly trace back 8 out of the 10 fine tuned models with our best method.
arXiv Detail & Related papers (2023-06-15T17:42:48Z) - One-Shot Machine Unlearning with Mnemonic Code [5.579745503613096]
Machine unlearning (MU) aims at forgetting about undesirable training data from a trained deep learning model.
A naive MU approach is to re-train the whole model with the training data from which the undesirable data has been removed.
We propose a one-shot MU method, which does not need additional training.
arXiv Detail & Related papers (2023-06-09T04:59:24Z) - AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering.
The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch.
The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level.
The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
arXiv Detail & Related papers (2020-01-15T18:32:06Z) - oLMpics -- On what Language Model Pre-training Captures [84.60594612120173]
We propose eight reasoning tasks, which require operations such as comparison, conjunction, and composition.
A fundamental challenge is to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data.
arXiv Detail & Related papers (2019-12-31T12:11:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.