Controllable Context Sensitivity and the Knob Behind It

• URL: http://arxiv.org/abs/2411.07404v1
• Date: Mon, 11 Nov 2024 22:22:21 GMT
• Title: Controllable Context Sensitivity and the Knob Behind It
• Authors: Julian Minder, Kevin Du, Niklas Stoehr, Giovanni Monea, Chris Wendler, Robert West, Ryan Cotterell,
• Abstract summary: When making predictions, a language model must trade off how much it relies on its context vs. its prior knowledge. We search for a knob which controls this sensitivity, determining whether language models answer from the context or their prior knowledge.
• Score: 53.70327066130381
• License:
• Abstract: When making predictions, a language model must trade off how much it relies on its context vs. its prior knowledge. Choosing how sensitive the model is to its context is a fundamental functionality, as it enables the model to excel at tasks like retrieval-augmented generation and question-answering. In this paper, we search for a knob which controls this sensitivity, determining whether language models answer from the context or their prior knowledge. To guide this search, we design a task for controllable context sensitivity. In this task, we first feed the model a context (Paris is in England) and a question (Where is Paris?); we then instruct the model to either use its prior or contextual knowledge and evaluate whether it generates the correct answer for both intents (either France or England). When fine-tuned on this task, instruction-tuned versions of Llama-3.1, Mistral-v0.3, and Gemma-2 can solve it with high accuracy (85-95%). Analyzing these high-performing models, we narrow down which layers may be important to context sensitivity using a novel linear time algorithm. Then, in each model, we identify a 1-D subspace in a single layer that encodes whether the model follows context or prior knowledge. Interestingly, while we identify this subspace in a fine-tuned model, we find that the exact same subspace serves as an effective knob in not only that model but also non-fine-tuned instruct and base models of that model family. Finally, we show a strong correlation between a model's performance and how distinctly it separates context-agreeing from context-ignoring answers in this subspace. These results suggest a single subspace facilitates how the model chooses between context and prior knowledge, hinting at a simple fundamental mechanism that controls this behavior.

Related papers

• Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting [15.69952375347308]
  Language models have the ability to perform in-context learning (ICL), allowing them to flexibly adapt their behavior based on context. We study structural in-context algorithms in a simple part-of-speech setting using both practical and toy models. We find that active forgetting, a technique that was recently introduced to help models generalize to new languages, forces models to adopt structural in-context learning solutions.
  arXiv  Detail & Related papers  (2024-05-28T21:38:20Z)
• Context versus Prior Knowledge in Language Models [49.17879668110546]
  Language models often need to integrate prior knowledge learned during pretraining and new information presented in context. We propose two mutual information-based metrics to measure a model's dependency on a context and on its prior about an entity.
  arXiv  Detail & Related papers  (2024-04-06T13:46:53Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models [58.42146641102329]
  We develop a novel semi-parametric language model architecture, Knowledge-in-Context (KiC) KiC empowers a parametric text-to-text language model with a knowledge-rich external memory. As a knowledge-rich semi-parametric language model, KiC only needs a much smaller part to achieve superior zero-shot performance on unseen tasks.
  arXiv  Detail & Related papers  (2022-10-28T23:18:43Z)
• Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference [10.536415845097661]
  We propose a method for predicting the performance of NLI models without fine-tuning them. We show that the accuracy of the cosine similarity approach correlates strongly with the accuracy of the classification approach with a Pearson correlation coefficient of 0.65. Our method can lead to significant time savings in the process of model selection.
  arXiv  Detail & Related papers  (2022-02-21T18:10:24Z)
• Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models [62.28551903638434]
  We measure the impact of three different adaptation methods on the generalization and accuracy of models. Experiments with two models show that fine-tuning performs best, by learning both the content and the structure of the task, but suffers from overfitting and limited generalization to novel answers. We observe that alternative adaptation methods like prefix-tuning have comparable accuracy, but generalize better to unseen answers and are more robust to adversarial splits.
  arXiv  Detail & Related papers  (2021-09-07T03:13:06Z)
• When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data [84.87772675171412]
  We study the circumstances under which explanations of individual data points can improve modeling performance. We make use of three existing datasets with explanations: e-SNLI, TACRED, SemEval.
  arXiv  Detail & Related papers  (2021-02-03T18:57:08Z)
• Multi-Modal Subjective Context Modelling and Recognition [19.80579219657159]
  We present a novel ontological context model that captures five dimensions, namely time, location, activity, social relations and object. An initial context recognition experiment on real-world data hints at the promise of our model.
  arXiv  Detail & Related papers  (2020-11-19T05:42:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.